Stop Bleeding Cash: Smarter AI Spend for SaaS

0:00Greetings, everyone. I'm excited to welcome Sarah Wyman, founder and CEO at StackPackPack. Sarah, welcome to the show. Thanks for having me. Hey, before we dive in, and we've got a lot of topics, these are just hot topics for CFOs and founders too. But before we dive in, tell us a little bit about yourself and your company. Yeah. Hi, everyone. Sarah Wyman. I'm the founder and CEO of StackPak. We are agentic AI for vendor management. And what that means is that we have AI agents. They help you manage your vendors, your contracts, your renewal, procurement, budgets, really everything to help you get control over your spend.

0:35And more and more, what we're going to talk about today is how spend is shifting towards AI. Yeah. And I think, yeah, that'll be the flavor today because spend management's been around, but I think, and software management, but now I'd say a bit of a revival, especially now with AI and internal AI spend. And so, yeah, this for me, top of mind for CFOs, top of mind and just that bill that's coming due. But, yeah, let's start with the basics. We see, and especially I see this in my business, those anthropic bills that come in as I use credits, not just monthly, but every week. ChatGP, Claude, just all the AI tools we're using eternally.

1:12So what can companies actually track today besides just knowing, all right, I've got this$50 ,000 ChatGPT bill. But what can we do with beyond that today? Yeah. So starting with the invoice, it's pretty limited, is the short story. But at least on the invoice level, you can see spend by provider, whatever it is, the tool, not only LLMs, but also just any tooling now that is all pricing using tokens and credits. So you can often see the spend by model within that provider as well, which can be helpful. I think the invoice itself is limited, I think, as we know, right? And so what you can't see is why that money was spent, which employee, which team, which workflow, is there a business outcome attached to it?

1:54So I think the invoice is a starting point. And I think a lot of teams are stuck there right now. Yeah. Which we get the invoice, we pay it. That's okay. What am I going to do about this? So it sounds like really not too much we can do at the invoice level. It takes a little bit more than that. So let's, yeah, let's have a nice little teaser for the CFOs out there. So what can we track today? How granular can we get with these dollars? Because it sounds like we need more than maybe the invoice. But I know a lot of CFOs now, of course, like any other Intel U software, where actually we don't want to just decode Intel U software.

2:28We need a new GL account. But do we want to track by API key, model, department, by employee, by project? So what is possible today? Yeah, I think in some ways we're beholden to the provider. And so for some folks like Courser, which we love, they break down everything by API key or user seat. You can also get this level of granularity with enterprise plans, right? So OpenAI and Claude. However, they're quite expensive. And so we do recommend to start using those. Typically when you hit that 150 employee mark or where you just are spending a lot, whether your product is using a lot of AI or your employees are.

3:07So typically over 100K a year in spend, it's time to start investing in that. And so with that, you'll be able to see employee usage, which is great. Then the whole key then is, I think, getting down to then the department level, right? And you can't necessarily do that with just the provider alone, right? You need to bring in other layers of data, such as IDs, whether that's attaching the user to actually an ID using your HR system or your SSO login, and also bringing in the outputs, which we'll certainly talk about. But it's absolutely all about bringing everything together from you have the individual user in the system.

3:45You then can get to department level and then you can bring in the use case layer. Okay. Because sometimes it does, these tools are absolutely a little magic, like of how they're pulling this data, which, and that's why I want to pull that curtain away for CFOs. But you mentioned, all right, we're getting some data, but maybe you mentioned, all right, we've got a login we're using or different systems. So is it taking more than just, say, me vibe coding some app in Cloud that integrates to the Cloud console and pulls data out, but maybe it needs more than that? It sounds like maybe you're using other logins to try to get to a more granular level.

4:23Yeah, what we certainly recommend is, yeah, bringing to get attribution, I think is what you're talking about, to get to true attribution of, okay, I got this bill and I know that engineering spent X amount and marketing spent this amount. It sounds like that's what you're trying to get to. And in our opinion, it's very much a design problem as much as data. And so what you're really trying to do is just get everything under one roof is the way we think about it. And so maybe you have keys that are structured by employee. That's your first cut. That's great. Then you match that user identity to your HRS system or maybe your directory so that you can actually assign spend to those cost centers.

4:58And that's really how you get from one open AI bill to down to the individual. So you do need to bring in the identity piece. It also layers up naturally to a department. And there's probably only two places to do that. That's HRIS systems and your SSO logins. And so it sounds like what makes this difficult is each LLM provider is a little bit different. It sounds like what maybe Anthropic provides versus OpenAI versus others. Because I think the evolution for me seems to be, all right, we get the bill. We pay it. Okay, that's not enough. Now, just like other Intel U software, we get the bill.

5:32Okay, who's actually using this? So first, what departments? Let's get that spend to the right department, especially with budget season coming up. I need to know who's spending what and because they're probably going to be asking for more AI spend. And then it sounds like that next attribution layer is, yeah, we need to know employees. So it'll be interesting. We'll get into more than this, but there's, I think it seems like a maturity curve here to getting a department down to the employee level. And is that, it sounds like that's, again, it depends on the provider. and then also maybe it sounds like you're pulling in different data pieces to do this.

6:07That's right. Yeah, I think, like I mentioned, with the Cursors of the World, they're going to give you exact employee and exactly what their usage is, which is great. With the open AIs and the clods, you're in one of two camps. Either you're using a team plan and individual plans where you're not going to get this data or you're on enterprise plan where you are going to get this data. So if you're on an enterprise plan, you are empowered here. you can kind of pull all this data together. Now, if you're on a team's plan, which is the majority of companies, they're not going to give you granular level data.

6:39So what they are doing, though, is they're giving you overall spend for the organization within that period of time. Let's say it's a month and they will show you overages. So our approach has really been to take the overall cost of that team plan, attribute it to all employees equally, because what else?

6:59You So at least you can find your outliers, right? Where is there a lot of spend or maybe where there's actually no spend and you're starting to track folks that aren't using AI in your organization if that is one of your organizational goals. Yeah. And when you mentioned those plan like enterprise versus team, was that, is that Anthropik and OpenAI? Do they have the same names or was that, okay, both? Yeah. Similar enterprise structures, yes. Okay. So that I think is a big factor, it sounds like, which I didn't realize is, yeah, it depends on what pricing plan you're on. and what spend visibility they're opening up to.

7:34That's right. And then I think the other piece to call out, we're obviously all talking about employee usage, but there's also product usage as well, both in your external product delivery, but then also usage of, I guess we'll call it skills or agents internally as well that you can set up on the API level as well. Very expensive to do that. Definitely a different cost structure. So a lot of this is challenging as well because there's the APIs versus the employee plans. Maybe you're using the exact same product, but the LLMs are charging you in a very different way. Those APIs are economically much, much more expensive to use.

8:08So we can talk more about that. Oh, sure. Yeah. And obviously in your application, you've done a lot of work with just designing this, bringing this data in. So what should that CFO dashboard look like today with the data that's possible and pulling in all these data pieces today? today. What if the CFO had that ideal dashboard, what should be on that dashboard? Yeah, that's a good question. I think there's a beginner dashboard, more advanced dashboard. So the beginner dashboard, I think, should have total AI spend, spend by provider, if you're using across all these providers, spend by department.

8:44Like, ideally, you're rolling up those employees by department. And then cost per employee, normalized for adoption. So again, you're looking to understand what are the unit economics of every employee. And then obviously we want to look at this over time, right? So month over month trends. I think that's the minimum. One thing I see on a lot of dashboards that makes me laugh is what's the total token usage number? And frankly, I find that pretty useless. Every single platform has a different value. There's no universal value of a token. So we avoid that yet because it's noise and it's not a real currency at this point.

9:17That will be interesting to track, certainly. So that's the one. I think where things get more interesting then is if you're able to add in what is the model mix, right? What percentage of spend is hitting the frontier models versus the cheaper ones? And then we monitor adoption rate as well, meaning what share of employees are actually using the tools that you're paying for. So that's a good place to start because you can at least see, do you have a cost problem? Do you have a waste problem? Do you have an adoption problem, really just use it for diagnostics. And then obviously the ROI layer, I would say, comes later.

9:50Yeah. And then so I think this is the next big step in that maturity curve is, we have some idea around spend models or providers, models, departments, maybe some employee level stuff. But now for a CFO, I don't want to get always blamed of being backwards looking and historical working. Looking now, it's like, okay, controls and governance around AI. So how do we take that next step to know, okay, I can have a good feel for spend, but how do I enforce it? What if someone blows their department budget and employees going way over? How do we take that next step to be proactive and have frameworks around SPAD?

10:31Yeah. The AI policy, I think, is a thing that's coming. So the good news and bad news is that on each individual platform, most of them you can set limits for. They don't have department level limits, but they do have employee level limits. So you can set those, they'll go off quite often, then it's up to you whether you allocate more capital to that individual or not. On StackPak, you are able to do this on more a department level, a more strategic level, but individually, you are able to do that directly in those. We have alerts set up that when you're at 80, 90 % of budget, you can set that, you'll get alerted on that.

11:04So most of the providers do have some type of native control, which is great. But I do think if you want a unified policy that's across all of the platforms, you do need that layer that sits above the individual tools, just so you can see overall usage, right? And it doesn't really matter on just one platform. Yeah. And I was going to say, you know, the balance between do we need more software and less? But yeah, I could see that because if we're using Claude and Chat and Gemini, And I interviewed another founder who has this app that lets you pick from 600 models. Yep. That's going to be hard to go into each one and set limits and all that stuff.

11:42So it seems like, all right, with your app, okay, the layer over that, we can pull this all together. So we're going to jump into 10 different things, which eventually is a recipe for disaster, I think. And as we're talking to especially engineering leaders, they're telling us that, hey, we want our engineers to be able to pick whatever model they want. And definitely there's different models that are better at certain things. And so they really don't want to limit that. And we are seeing just in our numbers that there isn't a lot of, we'll call it loyalty, to these LLMs quite yet. And will there be?

12:13It will be really interesting, right? Everyone's talking about the open models now. And so you want that flexibility. So that's another reason to potentially avoid the enterprise plans. But of course, we want the data analytics for attribution. Yeah, it's interesting. We saw the open router was acquired by Stripe. I interviewed a founder. So if you're looking now, if OpenRouter is off the table, check out trustedrouter.com where he built something where you can pick between all these models. And he was saying like, hey, there should be thousands of models, right? This shouldn't just be a few players.

12:42So really interesting. But that gets to now model choice. And I even, I've been, it's funny, I've been using Cloudflop. Then I've actually done actually a lot more in chat GPT with the latest model releases. And I asked, because now we've got Sol, Terra, and Luna. I'm like, hey, which model should I be using here based on the works? I printed out this cheat sheet that it gave me, which is really helpful for the finance stuff that I'm doing. But should, is finance getting involved now with model selection, how that should be governed? Now, I think model selection that goes into our product line, still engineering, they're controlling that.

13:20But internally, do we need governance around model selection and model choice? Yeah, I think it's honestly one of the most underused levers in AI native or AI powered businesses today is having finance set policies around all of this. And so if you do have that clean API data, you can see obviously exactly which models are being used. And the cost differences are sometimes 20x between one model to another. And absolutely, we're encouraging all of our customers to create policies. And that can be we've seen it a few different ways. One can be, hey, standard tasks, they run on a standard model, complex tasks run on a frontier model.

14:00We've seen other people assign, think of them as, what would you have an intern do? What would you have an analyst do? And assign different levels. And on our side, we've said, hey, if you're building content, use Sonnet. If you're doing a very complex task, please use Fable. I think there's a bunch of different ways to slice and dice it, but super, super important to do. And I don't think anyone's doing it really at scale yet. So we're in the early days. Yeah, especially, say, dynamic model routing, which is interesting. And yeah, there's such a discrepancy or a wide variety of prices where I ask a simple question, all right, what was the cost of that with DeepSeek?

14:37What would it have cost me with Fable? And it was 99X difference. Fable's so much more expensive. So it just shows you that it's so important, especially as more and more models come out. But I'm curious, budget season's coming up. We're recording this in September and a lot of CFOs thinking about budget right now, kickoff budget season October probably. But budgeting AI for 2027. Any tips, tricks that we can think? And CFOs, I'd say, love frameworks and not just run rating expenses. I don't want to run rate my AI expense next year. I would love some rate volume or something that could give me maybe some science behind it.

15:21If the board asked me, hey, Ben, how'd you budget AI? I kind of just increased it 10%. That's not a great answer. So any tips and tricks there? It's going to be, it will be an interesting year. This is our first year. I don't think 2027 is going to be perfect, but you and I have talked about this. Like there's a bit of an equation that we've talked through. I think it's certainly looking at obviously headcount, looking at the adoption rate. And again, if you, for some at our company, it's a hundred percent for others, it's lower. Looking at that usage on an average across all employees. If you can get down to the department level, I'd really encourage you to do that.

15:54And then looking at model cost. And so your headcount is fairly known, right? So that's good news. Adoption, I think you should have a pretty good idea right now of what adoption is going to look like. The usage intensity, this is like how many tokens, how many sessions. That's obviously going to be your big variable, right? It's going to be different across departments and you're going to have super users in every single department. So ideally, if you're able to do, again, if you're at a large organization, you're not going to be able to do this. At a smaller organization, you can do it on a per employee level.

16:25Larger companies should do it on a department level. And then model cost. You and I were just talking about all the models that are coming. Overall, we should see model cost decline, but usage is going to increase. Harder on the model cost, I think, to model that piece out. Yeah, price is always changing, it seems like, in a bit. Maybe some price wars. of, I know you're right, I get those emails all the time. Hey, we launched this model and here's the price. Oh, actually we're extending the early bird price on this and it'll be the same price as this and just changing so fast. And it seems like companies who are really have adopted AI where every employee, it's like, hey, here's your cloud account.

17:02We use it. It seems like, I guess it depends on where we are within our company, but it seems like you would have to get pretty sophisticated, especially if this is taking up, I don't know, 10 % of your budget. That it's going to just, you're going to need more than maybe just a person times a rate. But like you said, usage intensity, who gets what? Maybe what's the min and max of their spend or what it should be? A bear case and a bull case. It makes sense. Knowing that you're likely going to overspend in this category because it is just so unpredictable. Yeah. And I think that's where those alerts are going to come in handy because nothing is when I, What made me a little nervous was always the first reporting package of the year, that January reporting package of actual to budget.

17:51Did we make mistakes? Did we miss anything material that just creates this huge variance and we're already off budget, which is, I think, every CFO's nightmare to start the year. So maybe, especially with AI, maybe we start the budget fine, but then we blow it out in March. But I think that's where those alerts and actual real-time usage comes in handy. It's not just set it, forget it at this point with AI. Yeah, we're in our offering. We are managing not only AI, but actually entire budgets month by month so that it's not a quarterly or a biannual event, literally in real time on the vendor up and department level, how the budget is tracking.

18:28Yeah. And one last question or two before we wrap here, but now everybody and I see this and I get my kind of news updates about ROI and AI ROI, and it's hard to prove and just numbers all over the place. So you work with a lot of companies. What are you seeing? Are you seeing, we know engineering with pull requests and maybe we're developing faster, things like that, but maybe for other departments, it's kind of hard to justify that. So what are you seeing with your customers or just emerging trends with ROI? Yeah. Yeah, first thing I'll just say is you're spot on that engineering was the obvious place to start.

19:05But we're hearing more and more from our customers that marketing and CIS, client customer support, sales are actually becoming power users even more than engineering. That's been number one, just really interesting to see. I think on the ROI side, the most important thing you can say here is just it's not going to be perfect. You can't calculate a perfect ROI, but ideally you can triangulate it all. And obviously we would all love to say X person spent Y on Claude and that translated into X of revenue. Million dollars in revenue. Exactly. We hope to get there, right? Bringing all the data together requires a lot of tagging.

19:37What I think you can do in the short term is pick two or three output proxies per department. You mentioned tickets closed. But beyond that, actually sales and marketing is easier, right? You have deals influenced. You can do hours that are saved on certain workflows, right? And look at whether all of that increased, you know, after AI was adopted. We found that absolutely it has. That's kind of your baseline. What would that work have cost without AI? And then what does it look like with, I guess, with that fully loaded headcount? Had you had more headcount doing all that work? That's kind of your floor.

20:11And now where I think this gets really interesting is you start applying in real time all of these outputs across the entire organization, right? and looking and seeing, okay, here's the dollars that are being spent. Here are the outputs that we're seeing. You can actually start getting to an idea of what ROI looks like. And yes, the holy last step is certainly bringing revenue in. Yeah. And I think that output point is so important because if we think about, say, I don't know, our sales team and how many maybe prospects I can talk to a week or close deals, but can we increase that output that they're normally doing that they're normally measured on we can still keep quality but that output goes up i developed this rev intel engine that looks at our mrs schedule and then identifies dormant opportunities expansion opportunities that before maybe you had to visually search do all this and that but now it's there right ai made that much faster and yep here you go here's your playbook and that output increases we can do more with the same amount of folks obviously evolving.

21:14But that question is going to come up where we've, it's interesting to see from, hey, everybody use it to now, oh, we're spending too much to now, okay, what? What's the ROI on this AI? And then maybe that's going to come up in budget season. But yeah, we could be a budget season or two, maybe away from really maybe scrutinizing AI spend like any other spend in the company. I think this year is all about understanding what's the fully loaded cost of an employee now, right? An employee, the salary benefits, and now all the AI and vendors. Again, it's not just every software vendor has also moved to very real pricing.

21:50So we have that. So one last question that I'm glad you brought that up. And actually, I'll plug one of my metrics here first as far as ROI, because I have this Rose metric. It looks at how much recurring revenue you generate for every dollar of employee and contract or investment. And now you can throw in AI spend or agentic spend. And if that metric continues to move up and to the right, then you are getting efficiency with that spend. But if it's not, if it's flattening out or declining, yep, there is no ROI there. So just one plug for the right. Google Rose metric and it'll come up. But you mentioned, okay, other software.

22:27I think this is also a lurking problem in our budget is we have other software that traditionally was seat-based or subscription that has now moved to hybrid platform plus usage, which is really not anything new. But what do we do there? Because now I think that makes it even harder. Before we just said, oh, this internal software spend, yep, let's just add 10 % or whatever, unless we know a big deal or renewals come through. But what do we do there when all this other internal software, the pricing model is changing to usage? How do we stay on top of that? It's not even just usage. It's also outcome-based or even like, hey, I want to hire this agent for this month, add it into my package, right?

23:05So it's going to get really complex. Our recommendation is that you need to have a solid platform in place to be able to bring everything under one hood, basically, and constantly be extracting the pricing. And so what we do is we take every contract, every invoice, and we're extracting the pricing. and then again, it's monitoring in real time the changes so that you can capture something. That's one of my biggest fears is, oh my God, did we agree? Did someone buy something? Did we agree to something that it's going to be very easy to purchase? I think as agents make their way more and more into software applications.

23:41Yeah, definitely. Yeah. Appreciate your time today, Sarah, and great stuff. I love the great conversation. This is, I think, every conversation that a CFO wants to have and appreciate your expertise, but some listeners may want to learn more. If they'd like to learn more about what you're doing at StackPak, where should we send them online? Yeah, please find us on our website. We're at www.stackpack.ai. All right. And I'll have that in the show notes and there's some links there into your LinkedIn profile, Sarah, if they want to connect with you. But again, I really enjoyed this. I think it's the perfect time right now to put these frameworks in place for upcoming budget season.

24:18So we look good in front of our board with all this AI spend. So again, thanks for coming on the show today. yeah thanks for having me thanks