AI Agents, Google DeepMind and the Enterprise ROI Gap

0:00The AI industry is spending hundreds of billions of dollars building out computing infrastructure. The models themselves are getting smarter, faster, and cheaper. And increasingly, autonomous agents are moving beyond answering questions to actually doing work on our behalf. But here's the disconnect. While the technology is moving at an extraordinary pace, many enterprises are still trying to figure out how to turn those capabilities into meaningful business results. So what's holding them back? Is it the technology? Is it the data? The cost and complexity of deploying AI? Or is it something more fundamental about how businesses operate?

0:34Those questions were front and center last week at Worldwide Technology's Business Innovation Summit in Cabo San Lucas, where we brought together leaders from Google DeepMind, NVIDIA, Palo Alto Networks, Morgan Stanley, Glean, Cognition, Sire, and a host of others to discuss where AI is headed and what it means for the enterprise. And there were some fascinating perspectives on the economics of AI, the growing importance of enterprise data, the future of autonomous agents, and the challenges of translating technological progress into real business value. So today, we're going to connect some of those dots with WWT's Mike Taylor, who's back with us on the AI Proving Ground podcast.

1:11Mike was part of the summit himself, and over the past several weeks here on the podcast, he's offered a number of perspectives that were reinforced and in some cases challenged by what we heard in Cabo. So we're going to unpack what stood out, where the industry may be headed, and most importantly, what enterprise leaders should take away from these conversations. This is the AI Proving Ground podcast from Worldwide Technology. Let's jump in.

1:42Okay, Mike, good to have you back here in the studio. We survived Cabo, which feels like a little bit of an accomplishment in and of itself. You had a pretty unique vantage point at the Business Innovation Summit, and you moderated a conversation with Glean CEO Arvin Jain and Cognition Senior Managing Director Neri Singh. But you also had a chance to spend time with a lot of these leaders outside of those sessions. We've talked about many of these topics over the past few weeks, but I'm curious just to start out, what resonated not only with you, but what you think resonated with the rest of the audience?

2:16Thanks for having me back. It was tough to take a week off, you know, but I thought the session with Michael Wilson and the group were fantastic. And some of the things that I think we'll be releasing in the coming weeks as well, coming off of the event, we were able to get some points of view directly from many of the speakers that were there. I think that the best summary, I would say, is a real sense of community around both the opportunity and the threat landscape that's emerging around AI. And to be able to start the event off from, you know, or digging into Nikesh Aurora's perspective from Palo Alto on what they're seeing, what our audience was seeing that was validated, you know, in terms of some of those comments around the speed, the number of novel threats and zero days that were coming in.

3:07And it created, I felt like, this sense of community throughout the week, you know, in terms of sharing information with one another. In some cases, in a therapeutic form, in other cases, a little more productive in terms of how we take some of that on. But really, really thrilled with the lineup of speakers that we had and the connection between how do you look at something where it can be both a threat, but also just a huge opportunity for us and for our customers and the market in general. Yeah, I've got a number of thoughts that I want to get to as well. But before we dive in there, was there anything that surprised you or maybe challenged, you know, your current state and thinking on where AI might be headed?

3:44You know, there were a couple of things I would say from an instinct standpoint I felt strongly about or had been debating. One of which was this idea of, you know, you have companies who can, you know, feel self-serving in a sense of, oh, everybody's going to have a private AI deployment and this is what it's going to look like. And that's not just NVIDIA. You've got AMD. you've got Intel, you've got Google now with their TPUs. We really, I think, have moved from this idea of how do you go from a chatbot to an agent to now how do you go from an agent to agentifying a business process? So the term we were using was applied AI.

4:22And most of our customers last year were very much in the, hey, we're using chatbots. Maybe they developed some prompts that you would say were sort of the birthing of an agent, you know, kind of further down the path. But the number of companies who are agentifying or in the process of agentifying business process workflows was number one. Number two was the adoption of AI coding assistants and just how pervasive that has gone from this center of software to now a broader, you know, AI engineering concept. There's still so much to do. I mean, it's companies that are doing this in one or two areas of the business process and just getting out of maybe the software development groups now into the security operations, network operations teams, some of the infrastructure and cloud platform teams that are ultimately trying to get to a machine speed faster to, you know, to bring their businesses along and also defend that threat landscape that we kind of started the podcast off with here.

5:24So definitely moving from left to right. I think, you know, we feel a deep sense of responsibility to our customers is how do we move there with better precision, but also faster. And the early signs and indications are strong. But the piece that I'll add to it is my certainty now that we as a collective, you know, will have on-premise architectures that run models for us. Could be open weights, you know, open weight, open source models that are assisting with that agentic workflow or even the coding assistant work that some of the teams are doing, that feels just far more certain to me today than even some of the discussions we've had on the pod here in the weeks prior.

6:10Yeah. I mean, one of the things that stuck out with me in particular, and something I think actually ran through several of the conversations we had, was this growing gap between what AI is becoming capable of doing and really what the enterprises are actually getting out of it. We actually heard some pretty different perspectives on the pace of AI adoption. Nikesh Arora, who you already mentioned, the Palo Alto CEO, you know, he had one of my favorite lines of the whole two-day event. He argued, and this is a quote, we're probably overestimating AI's impact in the near term and underestimating what it's going to do over the next five to 10 years.

6:44He suggested, you know, consumers may actually drive the bigger disruption before enterprises do. And then just to add on that, Adam Winter, he's the CEO at Grok, He made a related observation from the infrastructure side. He said, much of the demand we're seeing for AI compute today is coming from AI native companies and frontier model developers. And that traditional enterprise demand that everyone anticipates hasn't fully materialized yet. And then, you know, finally, I'll bring in here in a second, I promise. But then there was Arvind Jane from Glean who shared a particularly revealing stat.

7:1575 % of enterprises are seeing individual productivity benefits from AI, but only 13 % report meaningful business outcomes. To connect that to a little bit of the current news landscape, Microsoft just released new research with IDC showing roughly two-thirds of organizations now report substantial or transformative AI results, but fewer than half say they're scaling that value across multiple business units. You know, we're seeing evidence of meaningful progress, but also continued struggle to turn that progress into something repeatable across the enterprise. Mike, you and I have talked about this distinction quite a bit.

7:49You've consistently made the point that individual productivity is not the same thing as transforming an entire business process. You've argued you need to measure things like throughput, cost, the actual outcomes of that work. When you put those findings alongside what we heard from the likes of Arvin or Nikesh or others, are we looking at an enterprise AI value problem or is the bigger issue you think that organizations just haven't figured out how to scale the successes they may be seeing in pockets? it's so hard to do i mean you know that the the the most basic unit of value i think in this in this whole effort is if if we went to and you know you're you're a big user of these tools as well if i went to my development teams or some of the lead uh business super users or business process owners and said, I'm taking these tools away from you, it would be a mutiny.

8:45I mean, it would be an absolute mutiny. And I think that that plays into this idea that as individuals, they've really leaned in and they've adopted these tools as part of their own individual workflow, right? And then, you know, we talked about kind of what is the structure of a high-performing AI-led company and its CEO at the top, its line of business leaders that are leaning into and leveraging, you know, partnerships and knowledge within IT. But those business owners, some of our best use cases have been digging into an area like sales, operations, or finance and saying, we have 10 different people that developed a solution around the same opportunity or challenge.

9:30And it has, you know, it has a ton of upside. We just have to then take those 10 things and turn them into one or two that are really great. And that's where we use the term, how do you move from workforce, AI, individual, or maybe small teams that are generating a lot of productivity and insight and scale that into applied AI solutions where the components of that applied AI solution are actually interchangeable between sales, material, order operations, or even finance. And, you know, that that takes real time and business leadership to go into a team and say, hey, I know you all are doing this 11 different ways.

10:09We're going to find one way to do this. It's going to be the best way to do it. It's going to be different for 10 of you, but it's going to be the best possible way for us to go in and do that. And as businesses, we're not instrumented very well to measure individual productivity outside of use cases like, you know, call centers, quoting functions that we have within the organization, material movement. But as you look at more complex business processes, these are spanning across different groups and teams. And the KPIs that we and our customers manage, they're instrumented to take, I would say, relatively large, complex business processes and measure efficiency against volume, ultimately, in that.

10:51And so on the efficiency scale, it's going to be really hard to measure till we get to those agentified business processes, you know, and build those pieces out. From a grow the business standpoint, I think we have huge opportunity to look at where are we expanding lines of business? How are we able to go to market faster, in particular with digital assets? And we've talked about, you know, the streamlined development and prototyping processes that are leading to more disruptive innovation from within those enterprises. The prototyping to me, and we talked about this on stage with the team last week, you know, there was always this, you saw the illustration of like, you know, requirements gathering and the telephone game that can kind of deconstruct in that, right?

11:34You know, the classic one is somebody who they're trying to explain they want a swing to hang from a tree. And, you know, they go and they design this elaborate, you know, thing. It's like the ability to prototype this stuff so fast and be able to put something in front of somebody that they can say yes, no, or maybe to. I think in conjunction with the code generation and the integration of these tools is one of the most profound things that we and our customers are seeing value of today. Yeah, no, that is absolutely interesting. I also want to build on a few more things that I heard from Neri Singh over at Cognition.

12:07He talked about how the real opportunity isn't simply using AI to write code faster. It's really changing the economics of the entire software development lifecycle, you know, from planning and development through testing, reviewing, deploying, maintaining, and on and on. But if you buy into that, the bottleneck likely shifts to things like architecture, code review, and product management. So what Neri says, does that illustrate a bigger challenge that making one part of a business process faster doesn't necessarily improve the performance of the overall process? Or how should leaders anticipate where the next bottleneck is going to emerge as it's always kind of moving around?

12:45You know, I really enjoyed Neri's perspectives last week. You know, he's a practitioner. He's been on a bunch of different sides of this. And, you know, we've had for a long time technology that would move the bottleneck within a process to one area or the other. What we lacked was a streamlined or efficient way to connect those train stops together into a real value stream that we can take on. And with the generative technology and both the ability to interpret and also produce, you know, on the other end, it has changed the game in the software development landscape that I think has spilled over into the security information security space as well to where, you know, we're not short on business requirements.

13:34We're not short of new ideas within our respective businesses. What we've been short of is the capacity to, again, quickly prototype. And then beyond just the generation of code, what happens next? You have to unit test the thing. You've got to you got to promote it through environments. You've got to run security checks against it. And so what we're trying to do is help. And Cognition is a big partner for us in this space to say this is reimagining the entire software development lifecycle. And I'm pushing our teams really hard right now to say, how can we develop a prototype engine where you, Brian, you could sit there, you say you have an idea around, it could be a new app, an existing application that we have.

14:15How could you literally talk to a machine that would hear you out, ask you questions and produce a working prototype? And when I say working, that could have access to enterprise data. That data has a context layer to it and get from idea to kind of functional prototype a lot faster. I'm pushing our teams. I think that is here and now. And with platforms like Cognition and others, including the frontier models that sit behind them, this to me is a real possibility in the next three or four months from an internal worldwide perspective and something we can share with our customers. Yeah, Mike, I mean, so much of what we heard last week, you know, it's circled back to the idea of data as the moat.

15:03You know, Logan Kilpatrick of Google DeepMind, he made an interesting point that the next major bottleneck for AI development may not be compute, but access to specialized data that largely resides inside enterprises. I don't think that's necessarily a surprise to anybody, but he went as far as suggesting that this trend could create an entirely new market for proprietary enterprise data. His argument assumes enterprise possess enormous amounts of valuable information. Mike, we've talked repeatedly here on this podcast about how fragmented, poorly structured, and inaccessible much of that information can be.

15:37Is the bigger opportunity here, do you think, is it owning proprietary data, or is it more developing the architecture and processes that make all of that data usable? Yeah, I do think it creates a new market. I think the discussion that we're having and our customers need to have is I think within that new market, these are conversations and decisions they've not had to contemplate prior. And the old adage of, you know, so what is so proprietary that it truly is a competitive advantage to that organization? And then you look at, OK, what is the opportunity to monetize the data between new product development or, you know, significant differentiation they can build?

16:18This is a new muscle for most of our customers. So I agree with Logan from a broad sense. How could companies think about monetizing the data that they have within industries they compete in or to adjacent industries that would be interested in the development of products or capabilities that consume portions of that data? The trend I see more companies today are perhaps recognizing that that is an opportunity in the future, but being far more cautious on wanting to keep, you know, the crown jewels of their data, if you will, within their own organization for right now. Yeah, another thing Logan mentioned, he talked a lot about benchmarks and specifically how companies can influence the development of frontier models by establishing benchmarks that reflect the problems they're actually trying to solve.

17:11You know, Mike, you've been pretty consistent in saying enterprises shouldn't simply adopt the latest model just because it performs well on a benchmark. You've been adamant you need to test it against your own workloads. You need to make sure it works in your own environment and understand how it performs relative to the cost. Interestingly, almost on cue, Google DeepMind this week introduced a new benchmark designed to evaluate AI agents performing realistic knowledge work. Rather than just measuring how models answer questions, it tests them on complex multi-step assignments involving documents, spreadsheets, technical diagrams, other materials that we all use every day.

17:46What's interesting is that the evaluation considers the quality of the finished work, while the research also reveals substantial differences in the amount of effort these different models expand to come to complete those tasks. As we move towards agents executing complex things within the business, whatever it might be, is evaluating the accuracy or performance of an individual model, is that sufficient anymore? Or should we be thinking more about reliability and economics of the entire workflow? Yeah, I think it's going to be a balance of both. And I say that for a couple reasons. There are going to be repeatable, high volume, repeatable tasks that are going to produce, consume inputs and produce outputs that will be used in a number of different areas of these businesses.

18:35And I almost think of those as like a bespoke machine, if you will, sort of like on the sidelines and it's running, it's consuming data, it's generating outputs, and those outputs go into a semantic data layer that can be consumed and used by agents that are performing, maybe in your context, I think we have to think of it in terms of process-based task assignment. And then you have what I'll call novel or creative-based task assignment, where it isn't a defined input, 10 different sort of steps, including evaluation and harmonization to an output, but the output or the outputs are generally not as defined in there.

19:21So I think on the machine churning stuff, like if you're a company that's processing hundreds of thousands of somethings today, I bet you have some machine learning algorithms already in there. You have some RPA you're already doing that with. Is there a better, faster, more accurate or more evolved way you could be doing that with generative. That's sort of the first piece. The second piece of it then is what we really haven't had outside of a great deal of custom software development are the idea of these agent to agent interactions and or call it a combination of recursive loops, where there could be fact checking or double checking, or just the ability for that process to recognize that it needs assistance or input from other sort of predefined or accessible agents to those.

20:09To me, those are likely going to include multiple models, even within that business process. And to start with, it may be, hey, it's just simpler if it's all this model. But over time, as we get more and more, we're going to continue to refine and look at those and potentially break them up into that. I think the other piece that is really important is the data gravity for this. So where is that processing going to take place? And more and more at the edge, we've talked about physical AI, robotics and other things that are going to be coming out. You know, you're not going to, depending on the parameters and size of some of these models, you're going to have to make decisions on just what models can you run, given, you know, the infrastructure, the connectivity and the latency that you have there.

20:55So I imagine cloud, on-prem, edge, all those. And then I think over time, we're going to have to start with a simpler set of models just to keep our sanity. But over time, it's going to be hard to resist not wanting to go in there and continue to refine both for the purposes of accuracy and cost. Yeah, Mike, I want to move on to the cyber portion of what we uncovered at the summit, because it certainly is something that's top of mind for pretty much anybody doing anything with AI. Nikesh Arora, the chief executive of Palo Alto, he was pretty measured about how quickly we're going to achieve true AI autonomy.

21:29He drew comparisons to autonomous vehicles and the challenge of dealing with all the edge cases that arise once you move from a controlled demo into the complexity of the real world. he raised a related concern about observability, how much visibility we're actually going to have into the reasoning and behavior of increasingly sophisticated models. Meanwhile, Yotam Segev, the co-founder and CEO of Sayara, he approached the problem from a little bit of data and identity perspective. He had a line that I thought was particularly interesting, which was control is a poor man's trust. His argument was essentially that you can't simply assume an agent is trustworthy.

22:06You need visibility into what it's doing, the credentials, the permissions it holds and the data it can access. We've all heard the comparisons of agents to interns or new employees, or we've even heard comps to teenage children. The point is, you need to give them clarity, clearly defined responsibilities and measuring their performance. But I think some of the larger questions that arise here is how much authority enterprises should actually be willing to delegate to these agents. How do you think about delegation to agents, not only within the four walls here worldwide, but of customers we interact with, what type of work can they take on knowing in the back of your head that the scary headlines about rogue agents are agents simply ignoring human input?

22:48Yeah, I think it's hard to ignore just some baseline patterns of how we establish trust. I mean, trust has to be one of the most, maybe outside of love, has to be one of the most studied emotions in human history, right? And it's going to take steps for us to get to that level. And I think we've said on the pod before these frontier companies, I think, think they're in the business of technology. They're in the business of building trust in terms of the methods by which they build these models and then the ability for us to sort of see into the non-deterministic world in which they operate. Right.

23:27And so I loved I love Yotam's quote, you know, there that one stuck with me as well. You know, in that. So what do we typically do? We even with the self-driving cars, I mean, there's patterns out there. I think the developers, again, are an early one with the need of engineering. We identify small tasks. We see how it performs with that. We provide some human oversight, and then we start stitching the tool chain together to make the task a little more complicated and complete. We provide human supervision. We relax that human supervision and go from checking everything to checking every tenth thing.

24:06In this world, we're going to have, I think, adversarial-minded agents that are going to be looking at some of this stuff and trying to help us generate where the challenges might exist. But a real-time example I'll give to you, Brian, is we've had a lot of zero-day vulnerabilities that have come out. And this was both internal and also working with a client of ours. We built an agentic capability to go in and identify where those vulnerabilities existed. These were some of the large, you know, zero days that were out there, the PHP. And then we had some Node.js stuff that came out too. And so we built a set of agents that went out and we're doing auto remediation.

24:47They're creating, you know, the remediation. They're issuing the PR. We were promoting those to our customers, dev and test environments and so on and so forth. And so the team continued to have some human supervision over this. Long story short, I think they, the first batch they did was like, 87 PRs in like a day. And the team's like, well, right now our bottleneck is we've got to go, you know, we've got to go look at every one of them and kind of go through this whole thing. I said, okay, fine. So go, go look through the first 87. I said, of the first 87 that you all went in and looked at, how many didn't work?

25:22How many were broken? And they're like, none, none. And I said, okay, so the, is the bottleneck going forward, human inspection anymore? Or are we confident that we have a semi-closed loop system here that is going to get this right 999 times out of a thousand, which is probably two or three times better than if we had a bunch of humans doing it? And that's going to be the trade-off here is to what extent are we going to let that loose and start to back away from that level of supervision? And everybody's going to have a different risk profile to it. But I think the tenets of change and trust are going to hold true in that fashion that I described.

26:10Yeah, I know we're running short on time on this episode, but I do want to bring up two final questions because I think they are important to the kind of the broader enterprise question here. The first being on leadership, I mean, it's become clear over the last several months that this isn't necessarily a technology issue. That's not the big limiting factor. At least one of them, culture and leadership and change management, they're major forces that are taking a part here. Logan Kilpatrick, the Google DeepMind, he raises a related idea from the software development side. He suggested that as AI dramatically reduces the effort required to create software, companies may need to rethink what their technical teams actually do.

26:47Instead of spending so much time writing software, the emphasis could shift to things like defining what needs to be built or managing the resulting products or continuously improving them, so on and so forth. I mean, businesses have spent decades, Mike, organizing people, departments, processes around the limitations of human labor and traditional software. If AI starts removing some of those limitations, it's reasonable to wonder whether organizational structures themselves need to change. It's a very large question here, but based on what you've heard at the summit, do you think we're approaching a point where traditional organizational structures become one of the biggest constraints on enterprise AI value?

27:27Yeah, I'll start. What I'm most excited and optimistic about is I am in more and more meetings today with teams where I cannot tell who's in IT and who's in the business. I mean, I know who they are for the most part, but they're speaking the same language. They're using some of the same tools. They're, you know, they're thinking about things differently. And And the barrier I really feel like to growth and innovation in many times is how do you connect technology to real business opportunities? And you think about from the business owner's mindset, they're running a business. They've got infinite demand from their customers and a competitive landscape.

28:06They don't have cycles to do anything other than perform in the moment. And the best of the best carve out 10 % or 15 % of their time to think about what should be next and how do we go and evolve. And then I take a look at, and you work with these folks, Brian, you know, our developer, our top development teams and individuals are those that understand the business almost as well as the business does. And they're thinking about, you know, and can relate to immediately the challenges that they're faced with. And so the more headspace we can give to those teams to sort of continue to come together and look, behave and sort of, you know, act and react in that way, the faster we're going to move and the faster we're going to be able to evolve the services that we offer to our customers and keep up with, you know, as Nikesh pointed out, you know, this wave of consumerism.

28:59AI that's coming at us, it's going to impact every single one of our businesses. Yeah, I want to get a final take here before we let you go. Try to bring a little bit of all of this together. So I mean, across these conversations, we heard some very different perspectives on where AI is actually headed. We heard that the models are becoming increasingly capable, but that proprietary enterprise data and context could determine who captures the most value. We heard extraordinary confidence in long term demand for AI infrastructure. But we also heard warnings that not every company financing that build out is going to succeed.

29:30We also heard enormous enthusiasm for autonomous agent alongside some pretty substantial questions about trust, accountability, and whether any of that is actually going to be prepared to delegate consequential work to these systems. You've spent a lot of time with these leaders, whether it was on stage at the summit or in private meetings, and you've had the advantage of seeing what enterprise customers are actually doing with AI today. Mike, as you consider everything we heard at the Business Innovation Summit and what you're seeing across the enterprise market today, where do you think business leaders are most at risk of making the wrong strategic bet over the next 12 to 18 months?

30:05Conversely, where do you see the greatest opportunity for companies as we move into the future? Yeah, I think there isn't enough emphasis I could put on the business leader from the CEO down to the line of business leaders to be thoughtfully integrating discussions around AI for the purposes of scale, differentiation, and market disruption into the flow of their weekly meetings, their one-on-ones, whatever the format is that they use to kind of run their businesses, there really isn't enough time for them to go and really dive in and understand this, but they need to be doing more of it than they are today.

30:50those that have created and even thought of and already acted on how do you how do you really bring the business and technology teams together and those are not just quarterly workshops this is like a real emphasis on sort of that that agile work environment where we're workshopping together i think co-location is important and being just being a real student and and and leading through curiosity, your own curiosity with your teams and disruptive thinking. I think Sarah Gellner, our EVP over marketing, who's also running some of our key AI initiatives, started the summit off by saying, are we thinking big enough?

31:30And I don't think we are yet. And that's the opportunity. No, excellent close out there. Certainly lots of opportunity ahead. Mike, as always, thanks for sharing your time here. Hopefully you have a good rest of the weekend and we'll see you next week. You bet. Thanks Brian.