#246 Will Granis: How Google Cloud is Powering the Future of Agentic AI

0:09I want to go build an agent. used to say good engineers copy great engineers paste one of the things that we're going to enable inside of agent space is an agent garden much like we have model garden today in vertex we're going to enable the same thing for agents so if you want to build a customer service agent if you want to build a marketing agent you want to build a data science agent you shouldn't have to start from zero so there will be a kind of a starter pack if you will of agents inside of agent space to jump start the development process create an oasis with thuma a modern design company that's specializes in furniture and home goods.

0:46By stripping away everything but the essential, Thuma makes elevated beds with premium materials and intentional details. I'm in the process of reorganizing my house and I'm giving Thuma a serious look for help in renovating and redesigning. Thuma combines the perfect balance of form, craftsmanship, and functionality. With over 17 ,000 five-star reviews, the Thuma Bed Collection is proof that simplicity is the truest form of sophistication. Using the technique of Japanese joinery, pieces are crafted from solid wood and precision cut for a silent, stable foundation. With clean lines, subtle curves, and minimalist style, the Thuma bed collection is available in four signature finishes to match any design aesthetic.

1:48Headboard upgrades are available for customization as desired. To get$100 toward your first bed purchase, go to Thuma. That's T-H-U-M-A dot C-O slash IonAI. IonAI all run together, E-Y-E-O-N-A-I. So for$100 off your first purchase, go to thuma.co slash IonAI. That's T-H-U-M-A dot C-O slash IonAI to receive$100 off your first bed purchase. Well, Craig, thanks for having me. Thrilled for the conversation today. What do I do at Google? So my current role is CTO of Google Cloud, which covers all of our technology, all of our customers, all of our industries, kind of a little bit of everything. And I came to Google via enterprise technology.

2:52So my background is building things, companies, technology, teams, all the way from being a founder of a one-person company, bootstrapped with a couple of Mac minis daisy-chained in a shared office space. living the dream all the way up to Fortune 10 companies and trying to build technology that helps customers, helps their customers. And from a background perspective, mostly in industrials, aerospace, manufacturing, the public sector. And then when I got to Google about 10 years ago, really spread out to cover all industries. So retail, financial services, banking, media entertainment, and all layers of the stack from Silicon all the way up to to SaaS.

3:35And, you know, current responsibilities, our team does two things. We work with the top roughly 200 customers around the world of Google Cloud. And the goal is to make sure that, you know, we build a lot of things and we have a lot of advanced technology coming out of the machinery here. But how it turns into value for our customers is not always obvious at the beginning. And so we help our customers get the most out of this advanced technology. And this, you know, this could been big query back when i started you know gke google container engine to auto ml to these days it's all about generative ai agents uh and eventually it'll be even quantum which we work on yeah yeah yeah the second bit is you can imagine working with these top customers all around the world uh we also learn a lot and we feel that the subtle tremors of what's to come and so our team also incubates engineering projects to try to prove the viability of a certain technology direction.

4:27And then we hand that off to the product team and say, hey, we've burned down a little bit of risk. We've got some subtle signals here. Here are some things you might be interested in. Then we partner to try to make some of those into reality. Yeah. Well, it's fascinating because you really have an overview of what's going on. We'd like to think so. It is a dynamic environment, Craig. One day you think that we're building on a platform for very domain-specific models and task-specific models in AI. And then the next day, we're working on kind of this orchestrator, generative AI, top-level engagement kind of mechanism where now a singular kind of model experience is orchestrating a whole bunch of those sub-models.

5:07And a shift like that happens in less than a year. So like the old school capital planning and all that has no relevance in a world like this. Yeah, that's remarkable. On agents, which is the flavor of the month or the flavor of the week. I don't know how long of the last. When did you guys start building agents with customers? When did you start seeing that as a trend? Well, I don't want to claim that we saw it so definitively that we mobilized everything we had against it. But I think back to the origins of even in 2017, my team was working with engineering and our internal incubator at the time was called area 120 and uh we we were working with a customer verizon and verizon has a really tough problem though they get a lot of inbound customer support requests every single day millions and of those millions of inbound calls what they're trying to figure out is how they can take the intent behind all of those calls and kind of cluster them into a smaller set of overall intents and then be more proactive in the engagement when a customer enters their funnel, it's actually a really tough problem.

6:23Clustering is a well-known method in data science, but actually proactively predicting the top trends or the top intents of your customer base is actually a very difficult problem to get right. And so at the time, we actually, with Verizon, took a little bit of this kind of intent clustering technology that we had built in Area 120, plus Dialogflow, which was kind of our agent assist, kind of predefined agent workflow tool in Google Cloud. And we smashed them together and we created the capability for Verizon to decompose all of those hundreds of thousands of intents every day into like the top five to 10.

7:05And so once you've got five or 10, you can actually proactively engage with your customers and say, well, what about this? And certain words and certain phrases convey intent. So it's not just like strict semantics anymore. And that was really, for me, thinking back on my 10 years now at Google, that was the first time that we really orchestrated an agent-based workflow. We weren't calling them agentic workflows at that point. We were calling them agents. But I think it's taken on a much more expansive meaning in today's lexicon when you say agent. So, you know, like you fast forward to today and, you know, an agent could mean you want to negotiate with a supplier and you send a procurement agent on your behalf to negotiate with an agent that represents that supplier.

7:50You actually are negotiating between agents in real time, you know, potential contractual terms or pricing, and you're embedding in these models, you know, like game theory or other mechanisms so that they can actually have dynamic negotiations with each other, that's much more expansive than what we were thinking about in 2017, for sure. Yeah. Actually, that's a fascinating example. I mean, what's certainly changed from the mid-2010s or late 2010s is generative AI, the transformer architecture. And these agents are now, they have conversational interfaces that make them a lot easier to interact with, but they're also now, as you just said, talking to other agents.

8:41I have to confess, I didn't realize that Google Cloud was so much in the solutions business. I thought it was primarily a resource provider. So can you talk about how agents have developed since generative AI became the dominant technology? Yeah, absolutely. There are a couple different directions that we should look at when you talk about the evolution of agents. You've already highlighted in my mind what is the most profound change today, which is we now have a natural language interface into every data store that you can imagine. And for those of us that have been in data and AI for a while, like I have, I mean, one of my startups, I was doing unsupervised learning on cyber events to try to cluster them and figure out where the most severe threats were.

9:41Being able to express in human language and intent and then be able to actually get business intelligence back short circuits so much of the data engineering problem that is, you know, for a data scientist and the data team, 80 % of their toil. That is the most profound change. And you could think about, I know we'll probably talk about AgentSpace at some point, which is our platform for building agents. But, you know, the vision behind something like AgentSpace is that it enables human language as the new programming language for AI. And so that by far is the most profound shift is this conversational interface to all of data and instructing AI using conversations.

10:22The second really big shift is the ability to now orchestrate an agentic workflow using an intent and actually being able to wrap APIs into tools so now that AI can go and access other functionality in a very seamless way. I'll give you an example. We are working with a customer right now. So it's confidential. My team works on a lot of very early stage stuff, but we're working with a very large customer that everybody would know. And one of the problems, this is like typical of an enterprise. There are really cool problems we're working on, like finding planets, new planets. And then there are other problems we're working on, like submitting your receipts to an expense system.

11:04So the problem we're working on right now is if you've ever worked in a large company, you know that you have to submit receipts if you travel or if you eat out or whatever. And it turns out that OCR, optical character recognition, has been a really tough problem for a long time. You have to train models. It doesn't usually get it right. And you've got all sorts of sadness. As someone who travels a lot, I've lived this firsthand. And we've been working with this really large company to say, well, what if we moved from typical OCR, which maybe captures the total on a receipt when you take a picture, gets all the rest of the stuff wrong.

11:39You have to have a human intervene anyway. And you end up doing like twice the work. to an agentic system that says it takes a picture and now it's first it's going to you know try to recognize all the characters but then it's also once it's got all the characters it's going to go to your audit system and say is there anything out of tolerance based on what it just has read you know is there anything that's out of compliance and then it'll also go to like a financial system and it'll say you know is this within the bounds of or is this what type of currency is this? What type of currency is you express them?

12:13So you have like five or six different API calls and tools happening underneath the scene. And all the user experiences is just a button press, an automatic cataloging of all this information and a submit button. And so that is, in my mind, the second big area of change is that it's no longer just a singular task executed with a very rigid kind of rules-based workflow. Now you're accessing other systems, gathering intelligence and wrapping it all in an outcome that a user doesn't have to like, and candidly, a lot of like analysts and data people don't have to like orchestrate all the bits and pieces too finitely.

12:51And the last bit is around trust. And you can imagine when you start getting, when you start imbuing agents with the ability to act on your behalf, it requires a whole of trust. And so, you know, we just talked a little bit about grounding. So, you know, we're grounding these workflows in, because one of the knocks against generative AI is that you got to be careful because it might not, it might hallucinate. Well, with grounding, you take that hallucination probability way down because now you're accessing authoritative systems, search company data, and you're reducing the likelihood of hallucinations, but you're also So you can now put guardrails on what these agents are allowed to do, and they can operate within a certain kind of rules of engagement, if you will.

13:38And then finally, you can secure this agentic workflow because now, you know, happening in cloud, you get to take advantage of all the cloud patterns like virtual private clouds and keep these systems from accessing the internet or leaking. And so, you know, you've got this trust component. So you've got this conversational interface, you've got this like rich tool and reasoning flow. And then you also have this trust foundation. But agent space is the, think about agent space as the singular platform that when you just want to build an agent, you want to index your data and tools to make it available to that agent.

14:11You want to provide the context. That's like the out-of-the-box experience. And it also wraps a couple of really important capabilities because we've always believed, and this is one of the things that brought me to Google, is the way to succeed as a platform business, which I would contend that Google and Google Cloud are a series of platforms and ecosystems. If you think about Android, if you think about Chrome, if you think about YouTube, now Cloud, these are platform businesses. And so as a platform, you have to be open. And so Agent Space is this nice kind of guided experience right out of the box.

14:46You get all these rich experiences. But if you've got another framework for building agents that you want to use, like LandGraph or something, You can use that and you can bring it to our Vertex AI agent engine and we'll take care of setting up servers for you and managing all of the identity access stuff. When you're trying to experiment, you don't want to have to deal with a lot of those things. And so it's easy for us using the native capabilities of cloud to spin up some servers, get you off and running. So you can use third-party frameworks. You can use ours. You can use our proprietary models.

15:18You can use open models like Gemma, which I think we just reached 100 million downloads of our open Gemma model in just the last year. But that openness is what builds affinity to a platform. And so whatever state an enterprise is in, maybe they're just getting started with agents. Or maybe they're really savvy and have been building a whole bunch of agents and they want to deploy them effectively, efficiently, and kind of build enhanced functionality. And there's a spectrum of uses of agent space and the underlying technology. Yeah. The example of the negotiating agent negotiating with another agent, you were talking about trust.

16:02Is that how autonomous would that be in a typical implementation? or does it then report, each agent reports back to its human minder that this is the deal we've come up with? Or are there cases maybe not that high dollar value or that critical where the agents can actually sign a contract or conclude a deal? Yeah, I mean, we're still on the frontier of multiple autonomous agents acting autonomously on more of the kind of high-value, high-risk use cases. Most of the use cases deployed right now are ones that you would find very straightforward. So, for example, BuildMeCreative for marketing. We're working with WPP, a world-class agency.

16:57and one of the things that they're doing is they're using gemini to be able to express an intent for a campaign and then within three minutes land on brand safe totally generative content that can be pushed out into social channels so that type of you know the agents like creating content refining it building it into a post and being able to deploy quickly with brand guidelines built in by the way like you can take the this is a great part about multi-modality in these models is you can take brand guidelines, you can just add it to the model as context without having to do like fine tuning. And it can adhere to those guidelines in the content that it creates.

17:34That's an example of, you know, what I would call like, you know, very current modern agentic workflow. We will see more and more of these autonomous multi-agent interactions, because if you think about it, you know, on a value curve and on a risk curve, you're going to take care of the low-hanging fruit first. So, you know, content creation and creative agents is one. Customer service agents is another. The other really popular one right now is code agents. You know, roughly 25 % of new code at Google is AI generated. And we're finding this to be the case more and more across our customers is that it's also, you know, these agents for code assist completion understanding.

18:16You know, those are things that can happen, you know, right now. But we'll see more and more of these multi-agent interactions on the newer use cases. We'll see more and more of that over the next few years. One of the really critical enablers for that is fast and secure agent-to-agent communication. And so there's a lot of work going on right now across frameworks and across protocols to make sure that they're passing the right information, they're authenticated correctly. And I would say like quite a bit of work right now is going into those platform level kind of identity authorization. And, you know, I mentioned the rules of engagement kind of guidance to agents.

18:58That's, you know, that's where we're spending a bunch of time right now to enable the multi-agent interactions. Yeah, that's, that's interesting. I had a call yesterday with a guy talking about self-sovereign identities for agents. Is that an area that you guys are working on at all? I mean, so that when agents, when you send an agent out into the world to do things, the counterparty knows that it represents who it's supposed to claim to represent, that sort of thing. Yeah, absolutely. And if you think about sovereignty and trusted execution environments, I mean, this is an area where Google has a lot of strength.

19:39We were just at Mobile World Congress talking about trusted execution environments, you know, that some of our mobile partners are utilizing to create new experiences on mobile devices. So it'll range all the way from agents acting on device all the way out to kind of these enterprise agents that are acting across systems of record. But that's absolutely our heritage. And in my opinion, a real strength of Google is verifying the trustability, if you will, of everything starting from the hardware that these agents are running on all the way up through the interactions that they're having and that they're taking and carrying that trusted identity.

20:19This also dovetails into another thing that we see in the future, at least and it's happening a little bit faster than even we had anticipated, is sovereign AI and sovereign compute more generally, which is we've now figured out how to provide our customers shells of compute and shells of AI with the controls, the data controls, the software controls, and the operational controls that best fits them. So like in Europe, there are very specific regulatory and compliance regimes for data locality, for example. Well, when you're dealing with AI, you have to ensure that as these agents are accessing tools and accessing data sources, that it falls within those sovereign controls and those regional controls.

21:07And so we have been deploying things like Google Distributed Cloud and Trusted Private Cloud to customers in Europe for the last few years. And that really sets the stage. So now the government can trust the data, they can trust the compute, and they can trust the AI that's layered on top of it. Yeah. You've been a proponent of multi-cloud systems where Google Cloud is, you build something and it's interacting with other clouds, non-Google Clouds. How do you ensure in those cases that the trust and compliance extends to the other cloud provider? Well, a lot of this is in standards work. A lot of this is in, you know, so like we put out a secure AI framework as Google, you know, but it wasn't just Google that was participating in this.

21:56This is a multi-vendor, multi-company effort. I think what you're going to see along frameworks and along protocols, you're going to see a lot more of this work be multi-vendor, multi-party. It may originate, think back to, like Kubernetes is a good example to me. Kubernetes was a technology that we developed at Google for managing a bunch of containers and compute effectively. But we open sourced it and we had, there was like this foundation, the Cloud Native Computing Foundation that was incubating this approach to managing containers. And eventually, all of the large hyperscalers utilized Kubernetes and managed Kubernetes services to manage fleets of containers.

22:39And it's just as popular across all three major hyperscalers. It's going to be very similar here. And so one of the questions that we're asking, for example, in our team right now is like, what's the equivalent of the CNCF for the technology that will enable agents and agentic workflows in the future? And again, protocols, frameworks, there will be a lot of work done to ensure consistency across the board. And governments and our public policy teams in support, we're looking forward to working with these organizations to define this future because it's not there yet. I think we just had figured out AI, and now we have these kind of autonomous agents as an extension of this AI.

23:27So it's something we all need to work on. Yeah. For building agents with agents based on Google Cloud, I mean, sort of the basic components are a reasoning model for the planning APIs and access to tools for, you know, making contacting other systems and making them them happen and then maybe vector databases or other kinds of knowledge um knowledge basis that that give some grounding is there anything else i mean are those the basic components or what what else is involved in building an agent well Well, so you've highlighted pretty much like the top level architecture. I would say there are a couple of nuanced things under the hood that are really important.

24:29So number one is if you want to enable great AI, data is really important. So one of the key things in agent space is enabling Google quality search on enterprise data. Because when this AI goes and looks for grounding or references, the quality of the data sources that it's calling will matter a lot. One of the emerging things that we've been doing in search and at Google for a long time, but our customers are now, the most sophisticated customers are utilizing is the combination of knowledge graphs and AI workflows. The relationships between things as as informative as the things themselves or the records themselves um so that's you know again enabling this google quality search uh over data is and these indexes that's critical for this um agentic workflow um to think about it from the point of a user uh i want to go build an agent what's the first thing software developers do when they need to go write code they go look for some code that already exists that kind of does exactly what they need to do.

25:43And as a friend of mine and someone I really admire a lot, Kelsey Hightower used to say, good engineers copy, great engineers paste. And so one of the things that we're going to enable inside of agent space is an agent garden. Just like we have Model Garden today in Vertex that I'm sure Nenshaw mentioned to you, we're going to enable the same thing for agents. So if you want to build a customer service agent, if you want to build a marketing agent, If you want to build a data science agent, you shouldn't have to start from zero. So there will be a, you know, kind of a starter pack, if you will, of agents inside of agent space to jumpstart the development process.

26:23And then from there, you know, it can get into connectors. So it's really important that, you know, data sources could exist. It could be everything from SharePoint to, you know, SAP to, you know, BigQuery. And so enabling the set of rich connectors, this is actually one of the things, having been here for a while, you know, BigQuery, one of the kind of breakout moments for BigQuery was when we were creating connectors to other data sources that could loop in and out of BigQuery more seamlessly and allowed people to do, you know, to use BigQuery for the analytics scale that it was really good for, but also loop in data more seamlessly in and out.

27:02And we're going to see the same kind of path in agent space, which is the connectors and the availability of those connectors is going to drive a ton of the value in the platform. It's just going to simplify the actual work of getting agents going. Yeah. And I'm sort of looking forward to where this is going. You know, I've spoken to Microsoft. They use the term society of agents. I'm not sure what Google Cloud's term is, but these vast multi-agent systems that will take over whole layers of an enterprise's operations, whether it's maintenance, tracking maintenance, predicting maintenance calls and things like that in a factory or something else.

27:55how big do you imagine these agentic networks can will get it or how big have you seen today yeah it's interesting i was just um i was just at a large customer of ours this week at their kind of annual tech kickoff speaking with their cto cdo this is a fortune five company and um you know the way i described it to them and the way that they're architecting it is that it will be a cloud of connected agents if you think about it that way. So it can scale up and it can scale down, just like the cloud. It can also be of extreme scale, but it can also be a job that just runs for as long as it needs to and is more ephemeral.

28:39So I like to think of it as a cloud of connected agents. And that's really the way they're thinking about it is there are going to be some agents that are long running, kind of always on and very active. And there are going to be a lot of agents that they will accomplish their goal and And then they will be, you know, we'll just knock the resource away and free it up for the next thing that we need to do and construct another agent on its behalf. And that's why I keep mentioning these protocols and frameworks and like the lattice work underneath, because it's all about scale. And you know, as a company, I think about another analogy is maybe like RPC and GRPC at Google, like machine to machine communication, the way for us to manage all of these, you know, you know, this vast fleet of machines and computer science operating it in real time is you have to strip away as much of the fluff as you possibly can.

29:28And you have to make these things as efficient as possible. So for example, like we care a lot about the packet size that's traveling over the network, right? Because today a workflow for customer service is a very, like it pops up, it happens for a little bit and then it goes away. Now imagine if you're running 24-7 continuous agent-driven ops for customer service. Your network could become very chatty. And so we're also thinking about how to optimize transaction sizes, payload sizes, all the little stuff that accretes over thousands, if not millions of agents. And that's the same way our largest customers are thinking about it because they're these diverse, multinational, many-function, um you know many vertical lines of business even within one company that's the way they're thinking about it because that represents you know the way their business actually operates so a cloud of connected agents is uh is how i view it wow that's fascinating uh and there's a lot of experimentation going on do you see these fortune 5 or fortune 10 or fortune 100 enterprises really implementing these into their operations, into these multi-agent systems?

30:46Or is it still people are talking about and experimenting, but the trust level is not there? I think the state of the art for the large enterprise right now is, you know, I mentioned the three common most common code, customer service, and creative. Those I see in most of the large companies and they're already executing even very complex workflows, agentic workflows against those three. What we will see more and more of, and I actually think this is one area where large enterprises may have parity with startups. Startups have the ability to move quickly in a very specific domain and like an agent for X and you just fill in the blank with X of like, you know, a business function or like, uh, what's an example might be, uh, you're trying to generate a document that, that, uh, in the pharmaceutical industry that has all of the knowledge and all of the state of prior attempts to get like FDA approved.