Real-Time AI and Multi-Agent Systems: Lessons from a Chief Architect at Visa

0:00Once we have these agents, they're going to be at scale. We cannot no longer have the humans in the loop to govern or guardrail them. Then the logical thinking would be like, oh, can I create another agent? Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI. You hear a lot about AI agents nowadays, but how you orchestrate and architect these agentic systems, along with considerations of human-in-the-loop, real-time requirements, and a ton of other factors become really critical.

0:37But lucky for you, I'm joined today by Praveen Gunasekaran, Senior Director and Chief Architect at Visa, alongside Hatchworks AI's own CTO, Omar Shanti. And Praveen brings a ton of experience from roles at Visa, Wayfair, and Microsoft, where he specialized in building reliable high-speed machine learning and AI systems for enterprise environments. But you guys ready to talk some AI? Yeah. Awesome. Awesome. Praveen, let's get it kicked off here. Just to start with basics for our audience, how do you define agentic systems from your perspective? Agents are anything which can automatically make a state change to your system in your business process.

1:23It's not just, we are not just talking about the chat-based information where you can retrieve information or you can understand some information or even sometimes help you to automate or autocomplete some of your tasks, right? But here we are looking at if that agent can make an actual state change or a business process change within your system, right? That has its own challenges because so far the couple of use cases that I talked about, the chat-based or even like, you know, able to complete something that you're doing, there is a new one in the loop, right? But when you're looking at the system at an automation level, business problems, there's generally not a human in a loop, right?

1:59Because you are looking at scale. So how can you build a system to be reliable, secure, and also available all the time is the bigger challenge here. Yeah. And when you mentioned, like, right when you said state change to your system, like, I think the hair is raised up on my back, the back of my neck a little bit, because with it, like AI and agentic systems, like they are probabilistic in nature, right? So that you don't always know the outcome. That's like the interesting thing. And when we were chatting earlier, Praveen, you mentioned this idea of like agentic thinking, like traditional software development.

2:36It's deterministic. I know every, or I should know, what scenarios are going to happen. You're developing the software for those. But when you're building an agentic system, it's different in that sense, correct? Absolutely, right? I mean, that's actually what's exciting, right? and why I'm really excited about the agentic system is before software engineering, it has to be very deterministic. I would even say that you have to stupid proof the system, right? When I mean you have to code everything, like about everything and everywhere about what that software has to do, right? That ends up limits what the software can do, right?

3:15And that's why coding or programming is hard because you have to look at all the use cases and you constrain them. Like lucky that the systems are constrained within your operations or whatever you're doing. So there's only a finite set of states available, right? And you code to it. But now, what that means like the number of people who are involved in it is, it's not many, right? It's really hard. There is a, the correct word would be like, you know, to get started with the systems, right? It's limited. But now, if you have a human-based conversation, so like the thinking is, okay, here is my problem statement.

3:52These are the bunch of tools or functions that are available for you. Now go figure out what's the best way to solve this problem, right? That's powerful, right? That means anybody who can understand this nuance, which most of us can, can get access to the use of computers, which is profound for me, right? Which is kind of democratizing your technology. That's why I'm very excited about this agent because it's the actual word I was trying to look for. It's like the entry to barrier to use compute, which is a powerful resource, is going to be really democratized and available for everybody. That's what these agentic systems and LLMs is.

4:29I think that's where the biggest revolution that's coming up, right? And that's going to give us a lot more productivity gains, a lot more people who can express, use the power of compute to do what they want to do. And we don't know what they will all do, but pretty sure they're going to do something good with that. Yeah, that's why I'm excited. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry.

5:06No fluff, no generic use cases, just real ideas that fit your business and the ranked by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder. I think the excitement level is huge. I'm curious, like Omar, from your perspective, like there's the one element that's like, oh my gosh, I can do anything with these systems and I can kind of just direct them, point them in the right direction. And they have the potential to execute on those tasks.

5:42But Omar, I guess the flip side of that is they could also do things that you don't expect, don't expect anti-patterns, like bad things as well. And the immediate thing is like, oh, I have to control this. I have to put in guardrails, which may not also be the best approach either. Omar, how do you think about that in a sense when enterprises are starting to like dip their toes into this agentic approach, agentic way of thinking? Yeah, well, the keyword there is governance, Matt. And governance is a set of muscles that enterprises have gradually become more and more likely to flex in the realm of data and AI, even in software development.

6:23Governance sort of couples the technology with the people and the processes, you know, in the triad that we're all familiar with. And some of the considerations of governance are around ownership, accountability, accuracy. and even as you're mentioning, Matt, sort of like putting some bounds and some guardrails in place to prevent worst case scenarios from occurring. I like to give the example of a call center transformation. I think the statistic is something like a call center achieves roughly 85 to 88 % accuracy. You could put a chatbot that might be able to hit those numbers. But the question is, what happens in the 12 % of the time when it's inaccurate?

7:05What guardrails are there? In the case of a call center agent, they're not going to agree to sell you a Tesla. They're not going to agree to give you a free car. Whereas, as we've seen publicly with some call center or some chatbots on the websites, when an error happens, it can be very costly and have major ramifications. So it's really important to think both in terms of the quality and the quantity of the errors that might occur and the negative sort of consequences and to put guardrails in place. and you can sort of adopt the same kind of prioritization matrix that you already use. So if you prioritize things based on customer satisfaction ratings, then certain things might be worthwhile to patch.

7:50Certain attack vectors might be more worthwhile to patch than others. We see something similar with data quality. An organization's got bad data quality. Where to begin? Well, with the most valuable data. And so over here, when it comes to governing your models, where should you begin? where the value lies. I think the other interesting thing too, I'm curious, both of y 'all here, there's almost an element of like fighting fire with fire, right? So if we unleash these new agents into the world that have this probabilistic approach to things, well, you could have agents that are tasked with making sure those things don't go off the rail.

8:28So that becomes its objective in a sense. So you may not explicitly know what negative things are going to happen, but you can have an agent that's watching out for those. Is that a logical approach? I'm curious if either of you have seen that in action. Is that a different way, I guess, of thinking, enterprises thinking about this? Definitely. That's an interesting idea. Because once we have these agents, they're going to be at scale, as I mentioned. You cannot no longer have the humans in the loop to govern or guardrail them, as you mentioned. Then the logical thinking would be like, oh, can I create another agent?

9:08We are already seeing the early versions of this. You would have learned about this concept called LLM as a judge, right? Where you are using another smaller LLM, like not that as powerful as the initial reasoning or the thought LLM has to. But the thinking comes from it's easy to verify a problem than to solve a problem. right? We can all understand and experience music, but we cannot all make music. Maybe now we can, but that's the thinking. Right, yeah. That analogy doesn't hold a physical nowadays with the generative AI, right? Yeah. Now you can have Suno.ai, which can make music for you, but still the creative process involves.

9:46So similarly, there are concepts, like there are some recent research I've seen where can we create other agents, right? A quality testing agent or a reliability agent right which checks how good the system is doing right how that specific unit of agents is safe another important concept like as you're alluding to more of orchestration right because with agents what you need to do is uh i feel you should not give them a really big problem because we are very early in the space right define them into really small unit task where it can where the complexity of the task is small enough so that you have a better ability the agents can like able to solve that in a better way and you can have other agent testing to start because we are very very early on this journey right you don't want to like jump deep into the ocean let's put your people to the water and wait it slowly and learn as we go so thus you have to put caution with how we are doing because these are powerful systems right you don't want to take this really powerful thing and start on the wild right you want to slowly wait into it yeah Yeah.

10:53So that's all right. So that's such a good thought process there, a principle. So listeners that maybe aren't paying attention or kind of zoning out, like come back to us real quick, because what Praveen said there, I think is critical. breaking it down into smaller elements that the agent is tasked with, right? Versus giving it this like autonomy to do anything and everything. If it's honed in on something very specific, it's almost like you're building a house out of strong composable bricks versus a house of cards that, you know, may get knocked over very easily. That's a really smart approach and way of thinking about it there.

11:31So just to piggyback off of that, there's so many directions to take that, but We've been long advocating on this podcast that the future that we're driving towards is a multi-modal, multi-model, multi-agent ecosystem. But let's build up to that a little bit. First, let's go back to what Praveen mentioned around LLM as a judge. So when this sort of advent occurred with generative adversarial networks, GANs, I think that's the right sort of elaboration of the acronym, you had a network training and testing another network. So from the very core of the architecture that brought us LLMs and these sort of genetic technologies is the same pattern that we can reintroduce to test.

12:14So LLM as a judge is swapping network test network for agent test agent. And we've spoken about this on previous projects as well. But if you want to think about the failures of traditional unit tests when it comes to a probabilistic system, you'll sort of realize that something probabilistic is best suited to test another thing that's probabilistic. you can sort of you can contrast the syntax from the semantics and by that i mean the meaning or sorry the individual letter to the meaning conveyed typically when we test software we have to test the individual letter the value of the byte but right now when we're working with agents a lot of the times we want the byte to be flexible that's why you know we don't all have our temperature set to the setting that minimizes variability we want that variability because that's sort of the root of personalization.

13:04But if you want that variability, you also need variability in your testing. And that's why agent test agent is a great pattern. But so to go back to sort of the broader question around how to test agents at scale and what this could sort of evolve into, one thing that we've been seeing on the market, Matt, is sort of a step to centralize in order to decentralize. So a lot of our interlocutors are building up single AI gateways that sort of connect the user to this graph of single purpose agents, which are each finally tuned to perform one task. It's sort of like this Unix moment, right, where you've got all of these functions that you can reach out to.

13:46And there's some form of communication so that these agents can invoke each other as well. So it starts to resemble almost like a service mesh turned into an agent mesh. if you will um so it's it's really fascinating the way that this field is heading yeah just to add to it where you know where omar was going is like we are going into a world where we're gonna have since you mentioned unix we're gonna have a newer way of operating system right an agentic operating system what that world is looks like because once you enter into this huge eag it's gonna orchestrate smaller pieces on it like all of these functionalities which you're gonna look require the smaller regions.

14:22You could have millions of them, right? And even in the world, the more writing code become very simple, easy, right? You don't have to build a software and keep holding and managing it. You can create things as you need on the fly and destroy it and use it, right? So that's an interesting concept that can happen too. So yeah, we are going to be where there will be a OS operating system we might see here. Yeah, it's a totally different paradigm and way of thinking of things. And I just keep trying to remind myself how early we are in this evolution we're experiencing right now. Like we're very early days.

14:58It's like back in the days of, you know, the internet, we're in the dial up or maybe even earlier of that in a sense. But one thing that's interesting when we talked last, you talk about these agents having different tools. It could be an API, which is how, in a lot of ways, computers and whatnot, they talk to each other through APIs or solutions and services and whatnot. CPU to run math or do something like that, a database, or even working with other agents. But you may have been hitting on this a bit just now, Praveen, but are APIs going to be that standard that persists? Do you see a new protocol potentially evolving outside of that, or does it continue to work in the same paradigm that we've been seeing?

15:46So I've been thinking about it, right? So initially, because APIs, all of these are defined by humans, right? The APIs and the message protocols and stuff. And I see earlier enough what I'm seeing that currently also, we're just going to use the existing interfaces, right? You call the APIs and just you're going to call them as an agent, but the agent is internally calling an API. or it could be even calling a function. But what I see as things, the more and more, we're not going to be building APIs long, as more as we are building now, that's not going to be a de facto way of building applications going forward.

16:21Now, you want to build in an AI native, an agentic native way of doing things, what would it look like? How would the interface be like? I mean, it's not completely clear. The verdict is not out yet. Is it like a chat-based interface or how do even agents talk to each other? Maybe the agents can come up with their own protocol, right? Because APIs are written for our understanding. The docs and things go out of the box, right? But if the agents, it depends on what optimization function you want to give this to the agents, right? For the problem to solve, maybe I directly converse in bytes or even like an other form of information, which it's more accurately, it can measure or reliably other agents can understand.

17:04So it's kind of fascinating that point where all this messaging protocol between agents is going to evolve. But for now, it's going to be, we're just going to put APIs and use them because most of the existing interfaces are available like that. Yeah, that's what I see, but it's very fascinating. Consider that REST APIs are simply abstractions over HTTP packets sent over a network. It's an abstraction that worked for a lot of the sort of single page applications and the recent turn to like web and mobile technology over the past 20, 30 years or so. But ultimately, there were predecessors and there's competitors even now, gRPC, SOAP, WebSockets.

17:46And within your operating system, there's all of these channels for inter-process communication so that you can trigger others and so on. So all of that's to say that at any point in time, there are dozens of these sort of protocols, communication patterns that coexist and that are in a niche. And I think I share with Praveen that the niche has yet to be discovered and there's no reason that it needs to be a REST API whatsoever when it comes to agents. Yeah, it reminds me almost of like how self-driving cars have emerged in a sense. They're kind of working within the paradigm and the way we drive cars today, but that may not be the end state if those actually start to proliferate and everybody is using self-driving cars.

18:28Similar here, it's working within the current ways of working that's been tailored to us as humans, but that potentially completely evolves in the future. Sure. But Praveen, when we were talking, you mentioned back at Wayfair, you were working in some solutions where you needed like 10 millisecond inference speed. And then you get into this concept of like real time. Do you need real time? And when you do need this like hyper real time, you know, sub millisecond type of scenarios. How does that impact your decision making from an architecture standpoint from, you know, leveraging cloud or not?

19:04What are the impacts there when you start to get to these crazy real-time type of solutions? So it depends on the business problem, what you need to solve, right? Because when you go real-time, as near real-time as they were talking about milliseconds, right? Not all possible you need to see, is there an AI solution as a right solution for it, right? Machine learning, I mean, inferences takes time and they are expensive enough. Okay, so first you need to understand that question. If it is, for instance, the way in Wayfair, the problem I was working on is predicting the delivery time. So it's a prediction.

19:42By definition, yes, you have to use ML. But can you use any algorithms, right? Probably not deep learning algorithms, like there is always constraints and computes on it, right? Once you have that understanding, then you will need to look at cloud, because what people will, people will work with, cloud will get you started very quickly. right but the stickiness of the cloud and the cost you will have to spend later i mean you might get some credits early enough it makes a lot of sense right too quickly but after six months or one year down the line your cost will not amortize right it may actually keep increasing so depending upon the scale of your organization right and the problem is it your core problem that you're going to solve right do you want to own the hardware by yourself or not right it's going to depend from business to business for example in in my current organization where we are big enough to uh own some of our the address we want right but in which where the use case was we they wanted to be fully on club so they were okay with that cost so there's no one solution but i have to be very careful about the problem which you are solving right what is the cost analysis of it right and is it something a core problem for it can you take this um the the architecture of the solutions which you are doing to any cloud.

21:00One of the ways I proponents in my current organization is because we run in multiple clouds, we have our own clouds. I don't just build cloud native, I build it cloud agnostic. Cloud native, there was a concept like, let's build it cloud native. But what it means, using systems very specific to that cloud, will keep you tied to that system. For instance, if you use AWS, if you use their PubSub, use their Aurora DB, and you're just stuck into it, right? If you want this to say, yeah, you can take it and move it into another system, another cloud, but it's not that obvious, right? Migration, we all see migration.

21:38Migration is never, never easy. It's the most difficult thing we all do. So the answer is complicated. It depends on the business, but a few principles I can say is keep it cloud agnostic, right? wherever it's required. Understand your business problem. If it is your core problem and if you can amortize your cost, the upfront RPC cost of the systems, go with it. And the third thing is build it in a modular and a composable architectural way so that the underlying technology, if they increase and make it better, you can always lift and shift, move your parts and be able to run on it. Yeah, such a good perspective there.

22:20And to sum all that up, it depends, right? at the end of the day it depends but you know you got to take that business context in there as well i'm curious i want to get yeah like spoken like a free consultant right it's another thought here that's why i'm like not worried about my job the he is not going to solve this job pretty soon yeah i think i think we got a little bit of bandwidth before uh before that happens for sure so i want you to riff on this topic we touched a little bit on this but y 'all brought up a really interesting idea around event-based streaming architectures, which that in and of itself isn't necessarily something new or novel, but then you bring agents into the mix where these requests may be streaming out there and it could be an agent picking it up.

23:04It could be something else. But I'm curious, like thoughts on that. And I'll let y 'all just kind of riff back and forth here for a second on that as an architecture approach or pattern, what that can mean, where it can be best used and whatnot. So I'll offer some thoughts first. I'd say thinking about the protocol is crucial because that's what sort of systems are built on. Systems are built on understanding what subcomponents within that system do, what they take in and what they return. We've been seeing this trend within the field of LLMs in general of supporting things called structured outputs, where you can sort of dictate what the format of the response looks like.

23:45That's brilliant because that starts to approximate the sort of protocol that Praveen and I were talking about earlier of having, you know, very clear contract, what you get in, what you get out. add in things related to SLAs, service level agreements, which can be like availability, time to execute or time to return a response, uptime, et cetera. And you start to more or less productionize a system around agents so that you can chain together these modular functional pieces without losing confidence in the system as a whole, because everything is governed by contract. I'm really interested in bringing this level of mesh-like governance to something like a multi-agentic workflow, because while a lot of our conversation focuses on the technology, this helps us sort of shift left a bit to the people and the processes.

24:41One thing that we can sort of take some, well, one sort of new exciting trend that we can glean into is the development of data contracts. Data contracts sort of stipulate what a data store contains, as well as how change management will be conducted. We want to drive towards a system of multi-agentic workflows. We need to see what the data product and data contract teams have been doing so far, pair that with what the service contract teams have been doing, and then find a nice synthesis unique to what you can achieve with agents in a multi-agent ecosystem. Sure. Absolutely. So if I have to add to what Omar was saying, right, because I feel for agents, an event-based system actually fits very well.

25:29The reason being is most, the agents are today, it's not going to be quick enough, right? They're fast enough. They have to do their reasoning. They have to do the planning. they need to try multiple times like a system has to work or they may have to do multi-turn or even multi-agent or process to come to a problem which means that you gotta give it a request and wait for it to come back okay which is kind of an asynchronous way and even just by by definition you don't have you don't expect a response out in the box right there right so the early adoptions of agents is not going to be on the real time systems right where there is you don't want to be uh using them for you know financial services when i swipe a card and there is an agent which is okay i'm going to think about it i'll tell you when it is brought you don't have that right people have less than half a second maybe less than that so then a swipe a card process happens but but you can take all these transactions and you can put an agent on the back end and the end of the day where settlement happens, right?

26:31At the end of the day, the process that an agent can take. So which means an event-driven system, a back system, which is nothing but a special case of your real-time operating, real-time systems, right? So which fits in very well to that architecture. So I see agents being good based on the event-driven architecture systems. That's where I see a lot of those developments are happening because you want your agents to actually think, you know, you want to give it a request and maybe, depending upon the complicity of the task. It's okay. I'm asking you to write a new movie and my characters are defined.

27:05I can sleep on it and come back in the morning and it is ready with it. That's great. Nice. I think that's a great stopping point right there, to be honest, because at the end of the day, it's about using the right tool for the right job. And that's all this new technology is right now is it's a tool to be used and you got to use it in the right way. But Praveen and Omar, Thanks for jumping on, talking some AI with us today. Praveen, where can folks find you if they want to reach out or learn more about some of the stuff that Visa is doing on the AI front? Absolutely. I mean, you can always connect me on LinkedIn.

27:42LinkedIn is the place I'm being more active on. Yes. Nice. Great. Awesome, guys. Well, thank you for joining today. Thank you so much. Thanks for listening to the Talking AI Podcast. If you enjoyed the show, give us a follow or subscribe on your favorite podcast podcast. And don't forget to leave us a review. We love those. For more info on Talking AI, visit TalkingAIPodcast.com. The single biggest mistake we see companies make with AI is they don't properly train their teams. We see it all the time. Companies roll out AI tools and expect people to just figure it out. But using AI effectively requires a totally different mindset and skill set.