941: Multi-Agent Human Societies, with Dr. Vijoy Pandey

27:50Jon Krohn:Cool. I understand all of that now. And I understand the analogy. It makes a lot of sense to me. Something that I understand is a big issue for agents today in this early ecosystem, this early internet of agents where we don't have all the protocols sorted out one of the issues that I understand we run into is identity and access management. So AIM is the abbreviation used for that often, which is kind of a fun thing because it's like I am, and you're providing your identity. You're saying who you are. So I am, that's a clever thing. I'd never noticed that before. And so traditional I am, internet or identity and access management systems often break down when applied to AI agents.

28:36Jon Krohn:Why is that? So identity and access management, by the way, I didn't even think of it as IAM, which is a great analogy there. But the reason the breakdown is, so far, all IAM services and systems have worked with humans or deterministic software. And agents are a combination of the two. So agents are like humans in the way they communicate in the way that they are using NLP, but also in the way that they act and behave and change their persona. So they're all probabilistic in nature. You cannot predict, I mean, you ask chat GPT the same question five times over, you'll get five different answers.

29:21And so they're very, very probabilistic in nature in terms of both intent and in terms of outcome. but they also operate at machine speeds and scale. So unlike VJoy, who, of course, is hallucinating half the time, who's unpredictable and is actually speaking natural language, but thankfully VJoy is not running at machine speed and scale. So if you give VJoy sub-access control to do some damage, you only give that access control for a short period of time. So you can think hours, let's say, or even days. Great. But agents are acting on behalf of VJOY today, might act on the behalf of John in the next minute, and they're doing this at machine speed and scale.

30:16And how do you build an identity and access management system to handle this probabilistic behavior at scale. And that is the big problem. So typically, the positions that we've used in IAM are three kinds of positions. Role-based access control, attribute-based access control, and relationship-based access control. These are the things that we've used in IAM to identify what access to give a human or a piece of software. and role-based access control works again because humans have certain roles and they don't deviate from those roles in a certain context like within our organization agents like we just discussed are not in the same time boxed role boxed environment and so they will differ and we

31:11Jon Krohn:need to treat them differently all right so okay now i understand why im systems break down for AI agents. And it totally makes sense to me there that a big part of the issue is that it's the machine scale, it's the machine speed that you can, if you provide permission to agents, you can all of a sudden have so much activity happening, so much network activity, so much real world impact in a way that a single human never could. So just kind of, it scales up the security risks associated with identity and access management. So yeah, what is the solution? So identity and access management for agents needs to go back to basic first principles and think about what is it that we're truly trying to achieve by giving somebody access to a protected piece of software, protected data object, or whatever it is that you're trying to give access to.

32:10So the basic principles behind that is looking at the task that you're trying to achieve, the tool that you're trying to access, or the transaction that is taking place. So we defined this whole notion of T-back. So T stands for task, tool, transaction, based access control. So that's the T back. Sometimes we want to call it T3 back because there are three Ts in this equation. But this is going back to first principles and saying, instead of trusting an agent by saying, this is an agent for VJOY, so let me trust that this agent is going to behave like VJOY for the next five days. Instead of doing that, I'm going to go back to the first principles of Zero Trust and say, VJoy's agent, what is it that you're really trying to achieve?

33:10Which task are you trying to do? And based on that task, I'll figure out what level of authorization and access you need. I'll give you a token to go and do that task at that level of authorization. and then I quickly bring you down before the next task or transaction or tool access takes place. So it's going really, really granular, figuring out what a task could be, elevating your privileges if needed for that task or tool access or transaction, and then quickly bringing you back because it's not just the speed, like you said, John, but it's also the probabilistic nature of the agent because it's not deterministic software.

33:55It's machine, it's software, and it's the data being fed into it and the probabilistic outcomes and the probabilistic communication that is taking place with the agent that makes it behave differently from one second to the other. Right.

34:11Jon Krohn:I've got you. Yeah. The probabilistic nature of the machines makes it particularly difficult to trust machines, because even if you actually trust the person who's provided this capability, probabilistic things, the probabilistic nature of LLMs means that you could get unexpected outcomes anyway, which could lead to security issues. So you have this zero trust agency ZTA framework. And yeah, and you have these two different types of access control. There's T-back, or as you said, T3-back, because it's task tool and transaction based access control. And then so that is what you were just describing, where based on some specific task or some specific tool or just one particular transaction that you need an agent to be involved with, with T-back, you're just providing control for that particular task, that particular tool, that particular transaction.

35:14Jon Krohn:And so that differs from the other way that you could be providing access control, which is role-based. And it's that role-based, R-O-L-E, that role-based access that is kind of, I guess that's traditionally what we were used to. You used to have like V-Joy would have access or John would have access or whoever would have access. that was role-based control. But now in this agency world, it makes more sense to have this T-back, which is much more specific. Yes, and I think the way to think about this is you still will have role-based access control for humans. You will still have attribute-based access control and relationship-based access control for software and humans and services that are deterministic in nature.

35:59But for agents, we want to move towards T-3back or T-back, task tool, transaction-based access control. And we need to bridge these two worlds because the old world is not going away. John will have role-based access control. Vijay will have access control, which is role-based. But my agent and your agent, and sometimes the same piece of software that is taking one persona at some point in time and some other persona at some other point in time. And so we do need to bridge the RBAC systems of the world to the T-back systems of the world for this multi-agent human societies to come together from the identity standpoint.

36:41I mean, if you're just looking at that narrow piece of the vision. And the other thing is, right now, everything that, I mean, there's a translation that's happening here, right? So I think you mentioned sometime in the past, but we all know this, where humans are going to be part of the loop for a long time to come. So even though there'll be agents in a workflow, in a self-forming team, whichever way you can think tasks will evolve or teams will evolve, humans will be part of the loop or human in the loop is going to be a thing for a long time to come. And so that's the other reason why role-based and task tool transaction-based access control need to talk to each other because there'll be points in this workflow where you will need to punt to a human and say, are we doing the right thing?

37:37Are the agents doing the right thing? Do you approve of this? And so that's the other reason why we need to bridge these two worlds together. But as far as agents are concerned, we have to move towards this first principle-based access control, which is T-back.

37:52Jon Krohn:Nice. That makes a lot of sense. When we're working with something like T-back, it sounds like it might be tricky. Like I can't wrap my head around exactly how permissions are granted just in time when you need to grant permission to an agent to be able to do a particular task and then you need to revoke that afterward. That sounds like it could be complicated. How do you handle it? We have to bring in a whole bunch of infrastructure around this notion of T-back to enable the end goal, which is the zero trust for agents, which is I give you permissions to do something specific for that just in time and then for that duration of that task to a transaction.

38:38And then I revoke that token or revoke those permissions the moment you're done. So what else do you need? So in my head, the equation runs like this, which is zero trust agency is I don't trust any agent and I just trust it once it's proven that it's supposed to do X, Y, or Z for the duration of that X, Y, or Z. And then I revoke those permissions. So that is zero trust for agents is a combination of the availability of trust, a task tool, transaction-based access control. So all identity providers, authorization servers need to support T-back. That's step one. We need to have a parsing entity for all communication that's taking place between agents and agents and humans that can parse that communication, that parse that discourse, and figure out the tasks or the tool access or the transactions that are taking place between agents.

39:44So that's the second piece, because that'll help us define what those tasks, tool access, and transactions are. So there's a semantic parsing element. And then your just-in-time comment basically implies that you get a token, you do that task in a very contained sandbox jailed environment, and then you're taken out the moment you're done. So the analogy I draw here is I want to access a safe, which has a lot of money, but I want to withdraw$10 from the safe. Now you can give me, since I have VJoy, you can give me access as VJoy to go and open that safe, and that's role-based. But then you can say, you know what?

40:34There are other people's money in that safe. So I'm going to give VJOY just enough to withdraw cash for the next 10 seconds and then move out. So that's like a task-based access control. But then I need to parse the communication that's happening between myself and somebody else where we are talking about withdrawing 10 bucks and say, okay, VJOY is allowed only to withdraw 10 bucks and that is the task he's doing. So let me just give you access for that$10 withdrawal and not sit around to withdraw all of the money from the safe. So that's the task-based parsing that needs to happen. And finally, I'll let you in into the sandbox environment, give you that authorization to withdraw$10, and then I'm going to shut the door because I don't trust you beyond that point.

41:25I will not let you linger around that safe. So that is a sandbox runtime environment that needs to happen. So is the hooks in identity providers to provide task-based access control? Is the semantic parsing of the discourse of the communication to figure out what that task is? And then a runtime sandbox environment to just do that task with that authority and then get out. So those are the three things that need to come together for zero trust for agents to have.

41:57Jon Krohn:Semantic parsing, ephemeral runtimes, and human in the loop approvals. And overall, you gave me a really clear picture now of what this all involves. One thing that I guess I'm still, that I still don't quite get logistically. You said that you won't trust the agent even for, you know, the particular task tool or a single transaction that you're going to approve it for until the agent has proven itself. How do agents prove themselves trustworthy in the first place? This is where the entire pipeline comes from the picture. So we are looking at identity, which is the first stumbling block that everybody is running into.

42:40And so one of the things that we're seeing is in the agency framework, the identity piece is the problem to solve first, even before you can start deploying agents at scale within the enterprise. But then, as you pointed out, there are other aspects of trust. There are other aspects of semantic parsing. So there are these other aspects of the entire pipeline that we need to solve for. So coming back to trust, the simplest way you can start with is saying, is there a directory somewhere that allows me to discover agents that are trusted? So I want to find a financial agent, not from a particular vendor, maybe, but also if there are 10 vendors, I want the best of breed financial agent from a vendor that is highly reputable and highly trusted.

43:35So the simplest version of the question, the answer to your question is, is there a directory which tells me which agents are trusted agents? But then the next question is, so that's where the directory comes in. So we have a directory where you can discover agents through capabilities and through things like reputation and trust. But then the next question would be, how is that trust enforced or attributed? Is it crowdsourced? Crowdsourced could be one thing. So 10 people have used it. is like the Apple App Store, which says five stars, trust, or trust pilot score. And it's like, yes, it's awesome.

44:17But if you want to be a little bit more mathematical and provide some rigor, then you go to the last pillar of that four pillar thing that I talked about earlier, discovery, identity, communication, and evaluation. You look towards evaluations. And you look towards evaluating agents and multi-agent workflows and saying, over time, I've built trust by evaluating this agent. And that trust can then feed back into the directory's reputation score and say, yep, all good to go. Till something perturbs in the system because these things are constantly evolving.

44:55Jon Krohn:That's cool. I get it. So this is kind of like when I'm on Amazon and there's some product that I want to buy. And one vendor on Amazon is providing the product for$5 cheaper than another. But the one that's doing it for$5 cheaper has a 70 % approval rating from people who have made purchases before. And the other one has 100%. I'm going to pay an extra$5. Right. And in that example, you and I who are buying this product are the evaluators. and Amazon is the discovery directory engine which has the trust reputation score attached to it because we as evaluators are actually feeding back into Amazon and saying, this thing sucked, so bring it down even further or this is awesome.

45:39Jon Krohn:So does all of this that you've now covered in the episode, so things like the Zero Trust Agency Framework, things like T-back, does that allow me as a data scientist or as an AI engineer now to maybe be able to get more of my agentic projects approved by enterprise security teams. True, and that is the whole goal behind this. When we thought about the Internet of Agents and agency in the open source, the notion behind the entire project was to ensure that agentic workflows and self-forming agentic teams and getting agents to do business outcomes or perform business outcomes, the bar to that is lowered.

46:22or becomes easier. And that's where we started. So identity is a big stumbling block and a problem to solve. Evaluation is a problem to solve. If you think about an enterprise like Cisco, we are dealing with agents from a sales force or from Cisco, Microsoft, ServiceNow. All of these things need to come together to build a simple sales funnel, for example. And so these agents are all sitting in their own clouds, different vendors, different clouds, different intent and outcomes. So if somebody says, give me the best X, what does best mean? When some agent comes back and says, here's the best X with 90 % confidence, is it 90 % well understood across all agents or is it a vendor by vendor thing?

47:11And so looking at discovery, identity, bringing them together and helping them collaborate and evaluating them. These are the four steps of the pipeline that we saw enterprises struggle with on a day-to-day basis. So to your question, yes, data scientists, I'm an engineer. We are all focused in on the outcomes that we want to deliver on our product, on the thing that we're working on. But if you take a step back and to your point, think about adoption. there are these really hard brownfield heterogeneity problems that we need to solve. And agency is out there to solve for it in an open source, open interoperable way.

47:56Jon Krohn:I like it because for me, I like to be able to focus on capabilities and I don't want to have to worry too much about security myself. of. And so I'm glad when folks like you and agency come along and you solve my security problems for me in a kind of, like a key, like, yeah, you just solve all the problems for me and I don't have to handle them. It's great. Turnkey. Turnkey is the term I was looking for there. Yeah. So it is an open source project. And one of the ways open source gets adopted and grows and is successful is if it is not a biased viewpoint. And so the way we think about agency and open source in particular is we want practitioners, we want developers, we want vendors, we want operators, we want consumers and customers, all personas and roles to come in, play with agency, figure out what's working, what's not, whether it's helping, whether it's not, is the documentation up to snuff?

49:00Any which way possible, help us grow that community, help us utilize the projects, provide real feedback. So even though you would expect, and we want to provide a turnkey solution, through the product stream that Cisco is building, based on agency, the open core part of agency, the open source part of agency, could benefit a lot from developers like you, practitioners like you, and the listeners here to come in and just contribute in whichever way that you deem necessary.

49:34Jon Krohn:Nice. And so if people want to get started today with agency, they like the sound of what you've been describing. They want to be able to have a more interoperable system for their agents to work with each other, to work with humans, and for this to work in a way that happens across whatever kind of framework they're using. You know, it works with Crew AI, it works with LandGraph, Llama Index, whatever. So, you know, obviously we're going to have the GitHub repo in the show notes for people to go to. Is there anything they should know? I'm seeing, for example, that there's something called Coffee Agency, which is a fictitious coffee company that helps developers, scientists understand how components in the agency ecosystem can work together.

50:19Jon Krohn:Maybe is that a place that you'd recommend people starting? Or, yeah, where should people get started with agency if they're curious? You can absolutely go to agency.org, and I'll spell it out again this time. So it's A-G-N-T-C-Y.org, and that's where you will get access to documentation. You'll get a pointer to the Git repo, and you can join and have fun. But like you pointed out, Coffee Agency is an open source reference application that we built. and it's truly a coffee agency. So we've got multiple coffee suppliers. It's a supply chain problem that we show. There are coffee manufacturers scattered across the globe.

50:58There's a full supply chain and then there's a coffee shop that's trying to actually sell coffee. And we picked this because A, we love coffee and B, this is a pretty complicated supply chain example which brings in all of the complexity that any enterprise goes through. And it's a reference application because it's a real application, but the entire code is open sourced. So you can plug and play various components within the application. So it's not just the agency components within Coffee Agency. Of course, it has A2A, it has NCP, but you can plug in Cassandra and see how that works. You can plug in, I don't know, Tipco has a messaging bus and see whether TIBCO will work as an interagent collaboration platform.

51:48So you can plug in various IDPs like an Okta or a Duo. And so this is one place where you can bring in your environment into this reference application and then swap things in and out. Maybe it's your code, maybe it's somebody else's code and see the benefits of using these various components and build out a real-world example before deploying it into production in your environment.

52:16Jon Krohn:Nice. Very cool. Thank you for doing this for us. Why are you doing this for us? Why does Cisco invest and all these other vendors? You mentioned tons that are involved in this agency project. Why, you know, what's in it for Cisco in the end? A few things. First of all, we truly believe in the open, interoperable future of multi-agent human societies. I mean, at least that's my belief. And Cisco is a big proponent of an open interoperable Internet and has always been. But that's a vision statement. That's a belief statement. There are business ramifications to it as well. And the business ramifications come, again, from the belief that open systems provide the maximal value for every participant in the ecosystem.

53:10So whether you are a vendor like us, whether you are an operator, whether you are a developer, whether you are a customer or a consumer, every persona in that ecosystem benefits if that ecosystem is open. From the dollar perspective, customers will get cheaper interoperable products. Vendors will make more money because this thing is widely adopted. So if you think about the value prop for every business, they will get maximal value if the ecosystem is open and interoperable. So that's the reason why we're in it. And also we feel that building this Internet of Agents is pretty much our birthright.

53:53And participating in this is something that we do well because we understand distributed systems, distributed computing at scale. and the next step towards distributed computing is actually agentic distributed computing. And so we want to go after that future and we can build it in the right way.

54:17Jon Krohn:I love that. Makes a lot of sense. Cisco are the masters of the internet. Why not be the masters of the internet of agents that it's coming? Makes a lot of sense. Before I let you go, actually, I was about to start getting into the final questions that I ask all my guests. But right before that, one final one popped into my head. Vijay, what's next for this initiative? What's next for agency? Agency so far, I mean, we touched upon this a little bit in the earlier conversations where we are building the scaffold for these different agents sitting in different organizations with different roles and personas on different clouds from different vendors to all come together to solve for a business or a consumer or a scientific or physical work outcome.

55:05The problem is that right now, whatever exists out there in agency and in other protocol layers and other frameworks like A2A and MCP and other frameworks that you mentioned, we are dealing with this heterogeneity, this complexity, this interagent collaboration at a certain layer. So we are dealing with what I call a syntactic layer of complexity. So this is like saying, agent one and agent two, you and I, I speak German, you speak Japanese. We are trying to figure out the structure of our language. And we just want to standardize on English. So we say, okay, noun before verb or verb before noun, this is how a sentence is constructed, and that translates to a framework.

56:00So whether you're built on a line chain or line graph, or whether you're built on a crew AI, or whether you're built on a bedrock, or the agent SDK from Google, these are frameworks, and these are just syntactical representations, and you need to make these things interoperate. But there's a bigger problem at hand. And the bigger problem, especially if we are true to our vision, which we are, we believe in this vision of multi-agent human societies, the bigger problem is to understand each other semantically. So even though we are speaking English, do I really, are we standardized on what we mean when we say certain things?

56:43So when we say best, when we say 90%, when we use a certain terminology or a phrase, are we standardized on that? And can we understand the meaning behind the communication that's happening? So the semantic layer is actually the next set of problems to go after. And when you go after the semantic layer, that's why I said there are many, many more protocols that will appear, but it's not just the communication part. The moment you start getting with the semantic layer, you're dealing with knowledge, you're dealing with cognition, as the problems get really, really interesting. When you take knowledge and cognition and you start spreading that around and you start thinking of it in terms of societies.

57:26So that's what's next when it comes to internal agents, per se.

57:32Jon Krohn:Nice. Getting closer and closer to that vision that you mapped out at the beginning where agents and humans are working together to solve the biggest problems facing society and making the world a better place to live in. I certainly share that techno-optimistic vision. Some people might think we're too optimistic, Vijoy, but I don't know. You can't stop the tech. It's going to keep coming. And there are certainly downsides to technology, but by and large, if I had to pick a time to be living in history, I want to be born now, not 100 years ago or any time before that. As a technologist, I mean, John, I mean, yes, every piece of technology has a downside or a negative aspect.

58:14But as technologists, I believe that we should build technology to solve for the problems that technology creates. So we need to solve it through fundamental principles and make things better. And that's just good business.

58:29Jon Krohn:For sure. All right, Vijay. So I already kind of alluded that this was going to happen. I snuck one last question in there. Every guest, I ask them before I let them leave the show for a book recommendation. What do you have for us? My favorite book is The Life of Pi. Oh, yeah. Life of Pi. Cool. And there is a big reason for that because I believe we live in a very probabilistic world. I believe that just like quantum physics, and that's the other thing that we dabble in day in, day out. Within OutShift, we're looking at the quantum internet and quantum networking. I believe that there is a superposition of states and we are just measuring and looking at one state and that's what we are in right now.

59:16And so you can believe in either science or you can believe in faith or a combination thereof. It's up to you. It's up to all of us. But whichever way you're leaning, there is a story that you believe in. There's a scientific story. that you believe in, there's a story around faith you believe in, or somewhere in between. That's a story. And the life of Pi to me was the eye opener in the sense that we all believe in stories. It's also like sapiens. I mean, we are a big believer in stories. But the life of Pi told me that believe in the story, which is more fantastical, which drives more excitement and which will make you live a more interesting life.

1:00:07Because you can always decide to believe in the boring story, which is what fun is there in that. So if you're going to believe in some story, believe in one that is going to drive excitement for you and just stick to it.

1:00:20Jon Krohn:That was a cool book pitch. I loved Sapiens. And it sounds like I'm going to now have to check out Life of Pi. So, you know, we're blending deterministic and probabilistic models to get the best outcomes for our enterprises, we can blend science and faith to get the most interesting outcomes for us as individuals. Cool. And yeah, very last thing, Vijoy, is how should people follow you for your brilliant thoughts after this episode? Obviously, we know to go to the agency website and the agency GitHub repo, which I'll have in the show notes. But what about you personally? Or are there any other links from Outshift or something that our listeners should be following?

1:00:57So I think agency.org was for agency. You can go to outshift.com. That's to follow everything that we're doing in Internet of Agents, but also the quantum Internet side of the house, which is even more fascinating and mind-blowing. but to follow me personally of course I'm on LinkedIn I'm the most prolific on LinkedIn so you can follow me on LinkedIn which is slash in slash vjoy on Twitter on X and then I have a website which is a little bit dated but I'm trying to update that but I think LinkedIn is the best place to follow

1:01:32Jon Krohn:Nice. Thanks for joy I agree that LinkedIn is the place where you can probably find most of our listeners and most of our guests these days it's pretty interesting how that's happened And great to have you on the show. I really enjoyed this episode. You are brilliant and fun to speak to. Thank you so much for taking the time out of what is surely a very busy day for you. This was a fascinating conversation. Thank you, John.

1:01:57Jon Krohn:In today's episode, Dr. Vijay Pandey covered his vision for multi-agent human societies where agents and humans collaborate to solve everything from scientific discovery to physical tasks, freeing humans to focus on creative work. He talked about agency, Cisco's open source platform for the internet of agents that enables agents from different vendors and clouds to interoperate and collaborate. He talked about the importance of blending deterministic and probabilistic tools rather than abandoning proven deterministic approaches for pure AI solutions. How digital twins of human systems in the physical world are key to building reliable agentic systems that can be tested safely before deployment.

1:02:33Jon Krohn:and the evolution from syntactic interoperability between agents to the harder problem of semantic understanding where agents quote-unquote comprehend the meaning behind their communications. As always, you can get the show notes including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for VJOY's social media profiles, as well as my own at superdatascience.com slash 941. Thanks everyone on the Super Data Science podcast team are podcast manager Sonja Breivich, media editor Mario Pombo, partnerships manager Natalie Zajewski, researcher Serge Massis, writer Dr.

1:03:11Jon Krohn:Zahra Karche, and our founder Kirill Arameko. Thanks to all of them for producing another exceptional episode for us today, for enabling that super team to create this free podcast for you. We're so grateful to our sponsors, and if you are ever interested in sponsoring the show, you can find out how to do that at johnkrone.com slash podcast. Otherwise, share, review, subscribe. But most importantly, just keep on tuning in. I'm so grateful to have you listening. And I hope I can continue to make episodes you love for years and years to come until next time. Keep on rocking it out there. And I'm looking forward to enjoying another round of the super data science podcast with you very soon.