What Your AI Spend Is Really Telling You

0:00Kevin Dana: Just looking at what we're experiencing, our token usage and our AI spend has increased from roughly$100 ,000 a month to over$1 million a month from the beginning of the year. And so the questions that we're wrestling with is exactly what you said there as part of your question, Brian, is what kind of value are we getting out of that? How much of it is fun experimentation? how much of it is rapid innovation, and how much of it is leading to enterprise applications that can be scalable, leveraging the AI tools that will be able to drive real ROI outcomes. And, you know, even though we've been at it a couple years, you know, I would say this AI journey, it's really hit what I would call escape velocity.

0:51Tom Strunk: In a matter of months, worldwide technology's monthly AI spending climbed from roughly$100 ,000 to more than a million. And even since we've recorded this episode that you're about to hear just a couple of weeks ago, that bill has doubled to$2 million. Now a CFO might hear that number and see a warning. An innovator might hear it and see momentum. And the reality is both reactions are justified because AI is creating a new kind of enterprise expense, highly visible on the bill, but often difficult to connect to an immediate return. While one employee may be saving hours, another may be building the prototype for the company's next major application.

1:29Tom Strunk: And a third may simply be burning through tokens without realizing it. So the question is shifting from whether enterprises should invest in AI to how they distinguish productive learning from expensive motion and how long experimentation can remain an acceptable return. On today's episode, we're talking with WWT Chief Financial and Data Officer Tom Strunk and Chief Information Officer Kevin Dana, who offer a candid look inside that challenge, what WWT is spending, what it's learning, and why disciplined investment does not always mean spending less. This is the AI Proving Ground podcast from Worldwide Technology.

2:05Tom Strunk: Let's jump in.

2:15Tom Strunk: Tom, I'm just curious to start out here. From where you're sitting, are AI costs out of control, or are we just learning right now what kind of the true cost is to scale this stuff?

2:25Kevin Dana: Wow, that is such a wide open and relevant question for where we sit just here today. But it's probably somewhere in between, if I'm being completely honest. You know, just looking at what we're experiencing, our token usage and our AI spend has increased from roughly$100 ,000 a month to over$1 million a month from the beginning of the year. And so the questions that we're wrestling with is exactly what you said there as part of your question, Brian, is what kind of value are we getting out of that? how much of it is fun experimentation, how much of it is rapid innovation, and how much of it is leading to enterprise applications that can be scalable, leveraging the AI tools that will be able to drive real ROI outcomes.

3:19Kevin Dana: And, you know, even though we've been at it a couple years, you know, I would say this AI journey, it's really hit what I would call escape velocity from a token's perspective in this past couple quarters, you know, with the onslaught of the improvement of the LLM tools, whether it's, you know, Claude or OpenAI or, you know, even some of the other models that Kevin may get into.

3:42Tom Strunk: The hyperscalers are certainly drawing a lot of attention for the billions of dollars that they're pouring into AI. I believe it's going to be a record-setting pace in Q2 of this year. Is that something that enterprise organizations that are consuming those services need to pay attention to? Or what is that signal about how they should be investing in AI moving forward?

4:02Kevin Dana: I think the biggest signal that that is, is whether you look at the hyperscalers, or what I would call the bigger players in the ecosystem, which include OpenAI and Anthropic and XAI, so the model providers themselves, and there's other pieces of the ecosystem as well. When you think about the amount of investment these companies are making, that is unprecedented and long-term. I mean, some of these investments in data center capacity, for example, these take three to five years to come online. And so for me, as I sit in my seat as CFO and as what I think about, it's important to our customers to understand, It is absolutely that some of the biggest, smartest, well-capitalized businesses in the world are spending years ahead on a technology that is generationally transformative.

4:59Kevin Dana: And you need to be part of the part of the ecosystem.

5:04Tom Strunk: Well, Kevin, I mean, it seems like most organizations around the world are abiding by what Tom is talking about right here. Gartner has AI spend this year topping two and a half trillion dollars, which is up a wide percentage from the year prior. I'm curious from like a CIO standpoint, how do you balance the need to kind of spend aggressively enough that you're competing without losing sight of being able to report back on what you can control or where the costs are going and things like that? It really comes down at the fundamentals are still there of business and IT or business and technology partnership.

5:38Tom Strunk: So in that regard, we want to make sure that the costs are visible and transparent. And that way, business leaders can be evaluating really the how on what they're being spent, what that money is being spent on. And to Tom's point, there's the aspect where a lot of the spend right now is in the workforce AI space, which is hard to quantify. But when executives are looking at that spend and identifying where there are use cases that can be translated into something that we can build and integrate into our systems leveraging AI, A, the overall cost for those become much cheaper, they become more predictable, and they become part of our overall flow.

6:19Tom Strunk: So it increases the efficiency. So in a way they play off of each other. And for now, we're pretty comfortable with getting a double benefit out of that workforce AI spend of educating our workforce and how to effectively use AI, as well as getting the ideas at more of the grassroots level. where are the missing pieces, where's that last mile that they used to have to do a lot of manual labor, that we can move that into our systems and extend that by using AI native engineering to accelerate how we deliver with the business on those systems as well. We've been certainly touching and talking about AI for many, many years now.

6:55Tom Strunk: But, you know, like everybody else, since ChatGPT exploded onto the scene, we've been doing that here internally at WWT. What have we learned about AI usage that maybe, you know, we didn't think about when we were just talking about it as a strategy. Did anything surprise you with the way the organization has adopted it or used it that would lend itself to the conversation of funding and scaling AI?

7:17Kevin Dana: I would say that up until earlier this year and the leapfrog of the newer versions of these models, we were on a rather predictable path of learning and experimenting. And I'll use one of our kind of showcase AI applications, RFP Assistant, that was one of the early ones. We started on that journey right after ChatGPT launched and whatever the end of that year was. All the years were running together. 2023. 2023. And it took us six months of back and forth and applying the model and learnings and ingesting documentation and all the things we were trying to do to figure out how to set up the infrastructure and have the models produce the right outcomes that we were doing.

8:07Kevin Dana: And we almost scrapped it. Six months, one project, dozens of people, and we almost scrapped it. A new version of the model came out. A model came out. And all of a sudden, the things that were broken started working. And for me personally, that was an aha moment that the models are going to catch up. The technology is going to continue to improve. So it's incumbent on us as an enterprise to understand and be connected to the ecosystem and have enough innovation and experimentation going on smartly to stay connected to it. And so that bridges us to a handful of other applications to earlier this year when, to me, another kind of step change occurred.

8:55Kevin Dana: Call it with the introduction of Claude, but, you know, the other models, OpenAI, have come out with their versions of it that are very good as well. and that launched it into the age of workforce productivity where we're now as kevin described putting these tools in the hands of not just developers who are using ai-enabled engineering or ai native engineering and and all the other coding tools that are making us much more efficient now it's in everyone's hands but for me the the acceleration of this journey over the last six months, both in investment and seeking ROI and looking for that pattern recognition.

9:35Kevin Dana: We can talk about data maybe a little bit, but the importance of the data architecture more so than ever. Not that we didn't know that 18 months ago, but when you're putting these tools in the hands of 10 ,000 team members instead of 500, it's a different ballgame.

9:53Tom Strunk: In today's world, with the state of workforce AI tools, anybody could conceivably create an RFP. But if you compare the cost of somebody doing that and the quality of them doing it on their own versus using the RFP assistant, the RFP assistant will produce what we want with the quality built in and at a far lower cost. So figuring out how we now blend it. So if somebody is in their workforce AI tool and saying, I need to create an RFP, that it would automatically detect and say, hey, let's start working through this agentic RFP assistant model as opposed to churning through a bunch of tokens is ultimately how we're trying to corral the best of both those worlds.

10:34Tom Strunk: Those use cases, identifying them, codifying them, but making it so that they're still accessible from that workforce AI cockpit that people are in. Yeah, that's an interesting discussion to talk about, taking use cases that may not have an immediate ROI attached to it or there's a little bit of a murky path to getting there. But RFP Assistant, like you mentioned, a great example of something that was maybe looked at as going closer to dead, but then the model changed and now it's revived and actually going in the complete opposite direction, providing a lot of value. As it relates to other use cases, Tom, I mean, how are you and the executive team, how do you evaluate these investments when ROI is not always readily available or the path to even get there is not immediately known?

11:19Kevin Dana: Very candidly, it can be a challenge. I would say we're working through it via extensive collaboration and a lot of rigor and discipline around what we're trying to accomplish. But on the spectrum of everything we're doing has a clear ROI to nothing, you know, we're probably halfway through the journey. And the other half that doesn't have a clear ROI, we are looking at as it's innovation. It's getting the tools into the hands of team members that are going to know their jobs better than leaders can and are going to help source the next set of great ideas. and and so when I going all the way back to the beginning when I talk about that spend in our ramp and where we are I've heard Jim say this and it's totally true our goal is not to spend the most per month you know our goal is to get the most out of that spend so whether it needs to be where it is today half that or four times that doesn't matter as long as we're eventually driving the ROI and we have enough line of sight and confidence to what we're already seeing that we will get there.

12:29Tom Strunk: How do you think about distinguishing between an innovative use case versus where you think we need to have a strategic bet? Worldwide does a wonderful job of establishing a culture of innovation. We have every year for as long as I've been here, and I'm sure before that, an innovation showcase. The difference, you know, pre-AI, is you'd have to get business people and IT people and dedicate time for them to come up with an idea, spend time working it through, usually get into something that's a rough order magnitude and then kind of pitch the use case and be like, this is good. Do you want to take it somewhere?

13:03Tom Strunk: And then that goes through an approval and then gets done. When you have something like the workforce AI capabilities, you think of the fact that you're able to get more into the hands of employees that know their businesses, as Tom said, and know where their pain points are and try something that doesn't require yet another person to come alongside and compete with, you know, our strategic priorities can still rough that out and then surface those and or find patterns of if there are five people that are saying that in a way that's voting of that's something that we need to solve. So if I have a look at that, that to me is almost like a strategic investment in the sense that we know we're not going to see a direct, directly attributable ROI for it, but it's expanded the ability for us to get that innovative culture out in and through that, the sensing to find that and route that into, as Tom said, the direct ROI where we can benefit both big and small.

13:55Tom Strunk: There'll be things that I'm sure we'll find, as we've already mentioned, some that are net new applications and capabilities that will have a tremendous ROI. There'll be other things that was like, there was just this last mile problem with our quoting tool and that PDF adjust. It's just like now we can actually get that out. And that takes out tons of hours and time and mistakes that people were spending on just feeding the application. So both are big. That improves the quality of life of our team members, improves our overall efficiency. And then the bigger ones are expanding our overall ability to execute as a business in the marketplace.

14:29Tom Strunk: Is there a line between spending on innovation for innovation's sake versus spending on actual tangible outcomes? Or is what you're saying that you never know when it's going to cross that threshold and start delivering, much like the RFP assistant?

14:44Kevin Dana: Kevin mentioned the term strategic priorities. So think of those as each team organized across the enterprise has certain business IT rollouts, tasks, new applications, things that are going to make us better, improve our customer experience, hopefully while doing it more efficiently. we have those strategic priorities that we're going to execute on. AI has helped make those go faster so we can do more. Kevin's team can do more. Concurrently, our IT organization and our business is learning a whole new set of these tools and it's happening really fast. So the amount of information coming at, not just IT, but the business is happening more rapidly than ever.

15:29Kevin Dana: So our executive team mindset is a little bit torn, but it is, we have to balance this. We have to make sure we execute on our strategic priorities while we're looking for those patterns and while we're innovating and reviewing workforce productivity usage and how our team's using it. And we may, just full disclosure to our team, and we talk about this openly, we may at some point have to put even more limits and throttles on how much people spend because we haven't been able to figure out how to get the ROI quickly enough. And we've already whet the appetite of 10 ,000 team members on what it can do.

16:11Kevin Dana: And so there is a lever and kind of a throttle we can move back and forth on how we do that. But right now, I would say we are over-indexed to more is better with the proper inspection and seeking the pattern recognition that I mentioned.

16:30Tom Strunk: So I think the other thing is, as these new models roll out, oftentimes they are more expensive. And they're the thing that, it's like anything in the human realm. It's like you see the new and improved, you're going to use that. So back to the fact of having broader limits, but then a new model comes out, the propensity is for people to use those. And so that creates that. So one of the things, and those tools right now do not have good mechanisms for playing back. What you just did cost X. So we're encouraging the innovation. We're also looking at how do we actually create more of a feedback loop of visibility for you to be like, here's your, here was your daily use and kind of a high level.

17:10Tom Strunk: You used it for some research, you used it for a prototype, whatever, and here's the price tag for that. So that helps to inform more responsibility for the privilege that they've all been given in order to use those particular solutions. So if we kind of take that element of all of these things, that helps to create the stewardship that goes with that investment. And then the other aspect of that is also looking at how some of these, the roles are shifting and things are changing. so a business person may be able to actually use a coding assistant to create a prototype but where's the right level for them to create something that's functional enough that then they can bring to it into the business and say yes that that fits into our strategic initiatives let's pick that up and carry it forward but we can do that with the level of rigor and discipline for something going into production right now we still see some of those where it's addictive when you start doing that they wasn't fill out the whole thing right so you spend more time money and energy producing something that we might still have to go in and kind of gut out part of it.

18:14Tom Strunk: You know, the prototype itself matches, but we have to fix those. So it goes back to the feedback loop. How can we teach people the right modes to go and express those things through innovation so we're not spending as much on full-fledged all the way down to the nines level of, it's fully functional, let's just deploy it because that won't happen. Yeah. This episode is supported by Snowflake. Snowflake's AI data cloud enables organizations to consolidate, govern, and analyze data across multiple clouds, powering AI, machine learning, and real-time analytics at enterprise scale. Unlock the full value of your data with Snowflake's unified cloud data platform.

18:56Tom Strunk: So in other words, you're talking about cost visibility just being paramount here, but also production visibility as well. I mean, I read a research piece here that said five times more likely, or KPMG was reported that AI operating costs, if you have visibility into that, five times more likely to produce ROI. Tom, do you think that that cost visibility, that's just a prerequisite now for success with AI in the enterprise?

19:22Kevin Dana: At this stage, absolutely. You must know, even if you can't trace that spend to a direct ROI yet, and it falls in the category of learnings, innovation, the things that I've already mentioned here, you must be tracing your spend. You need to know who's spending what, and you need to have a mechanism that reaches out to that person in a very, not a heavy-handed, what are you using this for? But a, hey, help me understand. You're spending this money. What are you getting out of it? How are you learning? Answer is absolutely yes. You need to know where your spend is, where your AI dollars are being spent and by who.

20:00Kevin Dana: And the other thing is you need a mechanism to follow up and look for those patterns and ask those questions. And so one of the things that we've done that I think our customers and partners might be interested in is the recent edition of Sarah Golnar's team, the AI transformation office. and we stood up that office as a centralized group to help bridge the gap between the individual business units and IT to help us have more bandwidth to go understand how are you using it and develop use cases out of those consistent patterns that can drive real ROI. And, you know, it's kind of amazing, Kevin, to me that we made so much progress without a more centralized group because just, you know, in the last eight to 10 weeks that we've stood this up, there's just so much collaboration.

20:50Kevin Dana: We're getting so much better visibility and bandwidth to be able to, to be able to draw out what those potential use cases are. And I know you're a big part of that.

21:00Tom Strunk: As the other thing with that too, just thinking, so younger listeners of this audience probably have not realized it, but the last time that there was probably a significant similar type of thing was when we were rolling out things like Microsoft office. And again, people probably I don't have to think about this, but we did have to teach an entire workforce how to use Word and Excel and PowerPoint at some point in time. Now it's just kind of ingrained and everybody learns it. But that would be a case of it's where business systems and processes intersect with personal productivity. And, you know, even then you see a lot of the same patterns of people pulling data out and using Excel to do whatever.

21:37Tom Strunk: Now, if you intersect that with workforce AI and needing to have a concentrated, here's the tools, here's the best ways of using them, with the interesting piece being that that actually becomes often a bridge between those very same office, you know, your office suite and your enterprise solutions, it creates those opportunities for identifying where the investments could be made to improve the flow of data and the flow of the process. The two of you now have mentioned workforce AI a couple of times. Can you just expand on what you mean by that? Is that different from solution-based AI or how are you thinking about that?

22:13Tom Strunk: We generally look at it as there's kind of three categories. Workforce AI would be your general purpose tools. You think of like a cloud, a codex, co-pilot, and the ability for you to use that as your personal agent plugged into the enterprise in order to improve your productivity and access to systems answering questions. We have our own internal one as well called Atom that we've built up from the ground up. The second kind of tranche would be your AI native engineering. Tom mentioned that briefly. And that's our coding assistant tools that are targeted specifically to your developer workforce to improve the overall speed, quality, and ability to produce solutions.

22:53Tom Strunk: And then the third one that you mentioned is really where you're embedding AI into enterprise solutions. So it's just part and parcel to the overall flow. And that's where, again, we see the best overall cost and the best overall discrete ROI. The AI native engineering, we're seeing there also tangible benefit because it's a specific slice of a workforce. You know what they're all doing. And then we're able to apply the processes and the improvement to get that done. The workforce AI is a little bit more open and that's where we're having to do more of the harvesting and the identification and discussions with team members on how they're using it in order to harness things that really flow back into the other two.

23:30Tom Strunk: The other thing we've discussed is the back-to-pattern recognition is as you've gotten it into the hands of people, we're finding that many of them are using it to recreate essentially the same dashboards. I mean, they're expressing themselves in different ways. But so that's led us to start to investigate how can we create the tools so that those reports can be published so that we're not actually turning tokens to create them over and over again.

23:56Kevin Dana: Yeah, it has two functions. One is the most common use case we're seeing, or a very common use case we're seeing is I had access to this data, but it wasn't quite complete, or this report wasn't quite complete. Also, I couldn't chat with that data. And so how do we serve up curated reports that are deterministic and secure that allows users to customize it, still get a deterministic answer via chat or a better HTML rendering? And so if you unleashed with our data, make this 5 ,000 team members, and they're each on the fly building their own report, you're going to churn so many tokens. But if you partner with your IT leadership and you say, hey, here's the outcome we're trying to drive.

24:45Kevin Dana: We have this group of people that's looking for this data and this type of report. We're going to push it out this way, and then we need to build it more securely and in a way that's token efficient. That's what we've learned just in a very short period of time. And it's pretty exciting to see the innovations and the revelations coming out of that effort.

25:08Tom Strunk: I saw a UBS piece of research that said 60 % of organizations are starting to put caps on AI spending. Kevin, I'm just curious as we've, the broader organization here at WWT has had its hands on Claude for several months now. What have you seen from a you should standpoint that maybe surprised you that gets us in the mindset of how to think about guardrails with tokens or where we're headed that way? We have some very exuberant users. We didn't go and say, what are you doing with that? But we're like, help me understand what you're doing with that. I think through that back to data, though, too, that helps us to also look at it over the time.

25:43Tom Strunk: and then through that kind of establish, we call it segments of users. So we won't go with the one size fits all, but we can find the right fit. This is, you know, general productivity seems to fit within this kind of profile. So let's kind of tune it there and back to the previous comment about educate through the AI transformation teams, et cetera, the what does that usage look like? How do you optimize that? You know, use it in the best ways and then step up into here's a different, you know, kind of class of user. Or maybe we have sales engineers that need to do things that are above and beyond.

26:15Tom Strunk: So then we would apply a different level for those. And then the key with all that is to empower the business leaders over those segments on what their limitations are, as well as the report of what users are using it for. And then the ability, if somebody has an idea that they think is going to go above and beyond, they can just have a quick conversation around, this kind of gets back to the ROI. Here's where I think there's a potential ROI. I'd like to prototype that and we can lift their limit for a little bit, let them kind of move into that sphere and then back down. So it doesn't, hopefully the structure is lightweight enough so that people don't feel constrained, but at the same time understand where those limits are settling so that they know if they're moving outside of the norm, so to speak.

27:02Kevin Dana: And this gets back to a little bit about the importance of visibility to your spend and the inspection. So we actually had a team member because they were spending a lot of tokens and we asked them what they were using it for. And they had some really interesting ideas and things they were building. But just explaining how much they were spending caused this individual to spend more to develop on their own an app that teaches him where his steps in the process are burning more tokens than others. And correct me if I get this wrong, Kevin, but he learned that by keeping six different sessions open and then coming back to them, every time he went back to it, it regenerated the context.

27:49Kevin Dana: So it burned more tokens, even though it was at a good spot before. So the lesson learned was just by that visibility and him taking the initiative to create an app that taught him, where am I spending more of my tokens? He learned that, boy, you shouldn't keep six sessions going open at the same time. You should finish one or maybe use one or two, and it'll cut your spend in half or more. And now that's a pattern. We've we I say the team is for certain individuals in that group that has that kind of usage. All of them are using that homegrown app that that person developed to help track their spend and give them more visibility on what steps they're taking that might be burning tokens without the outcomes they're looking for.

28:37Tom Strunk: Well, to the two of you, thank you so much for for joining the show today. That was insightful and effective stuff. So appreciate you joining.

28:45Kevin Dana: Thanks for having us. Yeah. Enjoyed it.

28:47Tom Strunk: All right. Thanks to Tom and Kevin for joining us here in studio. The lesson from that conversation, it'd be irresponsible to give AI a blank check, but demanding a guaranteed return before every experiment begins, that might be just as costly. The discipline lies between those extremes, making spending visible, giving employees feedback, recognizing recurring patterns, and turning the strongest experiments into secure, repeatable enterprise capabilities. This episode of the AI Proving Ground podcast was co-produced by Nas Baker, Cara Kuhn, and Marissa Reed. Our audio and video engineer is John Knobloch.

29:20Tom Strunk: My name is Brian Felt. Thanks for listening. We'll see you next time.