Measure Engineering Like You Measure Sales (w/ Dan Bender & Kirby Montgomery, Code & Co.)

0:00We were working on a project two weeks ago and the CTO is like, hey, my CEO read about a thing called a dark factory. A dark factory, for those who don't know, is basically vibe codes being written and you don't really know if it works or not. The robot said it was tested, said it works. Dark factory sounds like a great Marvel movie. Maybe John Bernthal is in dark factory, right? It's perfect. It's not venture capital. It's private equity. It's the private equity funcast. Pour yourself a drink and have a seat. Welcome back to the Private Equity Fundcast. I'm Jim Milbury, technical partner at Parker Gale, which means my partners ask me some version of this question every quarter.

0:41We can see sales pipeline. We can see finance down to the decimal. Heck, Paul Stenzik even has marketing measurable at this point. So why can't we measure engineering? It's a fair question, and I'm the one who has to answer it. So today I brought some old friends, Kirby Montgomery, Dan from Code & Co. Welcome to Chicago. Jim, thanks for having us. Good to be back. Always a pleasure to see you boys. Thank you for having us. And you guys have been flying all over the place. We were in your old stomping grounds, Boston. And Dan, I can say, ate about four or five lobster rolls. So I'm surprised he's actually here today.

1:13Look, I rolled in. Nicely done. And that's a huge, like, North Shore of Boston, they'll throw, like, relish on a lobster roll. I find that to be absolutely disgusting. You know, that's an abomination, right? South Shore lobster roll is hot dog roll with butter on the outside. They grill the roll and it's just lobster and mayonnaise, boys. That's it. So here's today's map, Joe, right? First, why has engineering been so hard to measure over the decades? Then today, what can we actually measure now that's realistic and mostly accurate? And then what this whole process looks like from running a deal all the way through owning the deal and all the way through selling it as part of that whole process.

1:57So last time, Devin, Lucas, and Dan walked through the scorecard and the questions you asked before you buy. Today, we're going to find out, all right, now we've wired the money. What do we do now? So should I give you like a real easy one to warm up with? Here we go. Ready? Softballs, please. I don't know that softball is all that hard. I would say wiffle ball, probably. All right. So before we start, what an investor says to you, I want to measure engineering. What do you think they actually want to know? And I'll start with you, Kirk. So overall, for an investor, to me, it always goes back to the thesis.

2:25They're buying a company and they're like, okay, what skeletons are there? We find that in the diligence process. And during the hold period, they want to know, can that system continue to match the financial plan? Is it scalable? That's always the biggest one. And the second one after that is, can the architecture withhold or build away from expanding that to them and these things are kind of squishy and we want to make them less squishy by saying okay this is a language that you can understand scalability and then also what's the cost of that that's where we get into like what is the impact of ai on a given engineering department these things there's tons of metrics for them engineers love to talk to other engineers about why a certain metric is wrong but to a non-technical person it needs to be simple easy to understand easy to pull the variance out of, and then also trackable.

3:14Month over month, quarter over quarter, how is that looking in the hold period? So Dan, I want to steer it slightly differently for you because you made an excellent point, Kirby, which is the private equity guys look at this business, they put a plan together. I'm going to tell you that most times the CTO of that company has no idea about what that plan is when that deal is closed. Is that something you find to be true? Yes. Yes, I think ultimately there's oftentimes a disconnect between, I guess, the commercial strategy, the value creation hopes and plans and sort of the engineering realities.

3:46And I think it's never been more obvious than now given agentic engineering is real. For example, suddenly the cost of a feature has gone down so much. On the other hand, you don't want to become this token furnace kind of factory, right? We're just like burned through tokens mindlessly. plus the total cost of ownership when it comes to software engineering is still unknown, right? There's some teams have designed these really good harnesses and designed really amazing and sophisticated coding factories, great stuff. Others, they're kind of like vibe coding, right? So I think this topic of communication and alignment between commercial leads, whether it's a CTO, a CEO, a CFO, and sort of the implementation teams led by the CTO and CPO has never been more important, but also more complex, I guess.

4:35Definitely more complex. So you said something funny, hope is not a strategy, right? So you know the feeling, you walk out of a board meeting where we just decided to put a big bet on an engineering rebuild or something like that, and everyone's quietly hoping that it's not a black hole, right? So Kirby, you're in diligence every week, right? Where does the gap between investors and engineering leaders really show up? I think it comes into some of those larger builds you're talking about. And so say, for example, on like a buy side diligence, someone's like, hey, we've got some tech debt, we're going to do a migration.

5:04To us, that's a smoking gun to say, okay, well, prove us the progress. Where are you at with this? How are you tracking it? And it might be a very large build, a small build, but like, have you built anything to prove you could actually do it? And then from there, what are your metrics that are making sure that everything that you do gets easier, faster, smaller to release? And so in this kind of scenario where you've got an investor, you have a tech leader, the biggest things are always true is, is the system stable? Can it scale? And then are the people working on it, working on the right things?

5:35And if they are, how are you measuring the outcomes of that? And if that's a disconnect, they don't know those answers together. We know the engineering team's probably not strategic. The leadership isn't involved in that space. And if they truly want to drive outcomes, or like you said, be part of a real value creation plan, then they have to have a shared set of accountability metrics that they can measure together. Also, I think the realities of engineering is very often the first 90 % of a project take 90 % of the time. The remaining 10 % take another 90%, right? So I think engineering has always been hard, right?

6:08And there are always, you know, things that change as you're working on it. So I think you just have to have a lot of empathy for the fact that not everything can be planned, but in order for, you know, learning whether something can be improved next time around, for example, you need some data, right? So this whole tracking bit, I think, is so foundational and important. And this is where we see different levels of sophistication across companies. You get a good point, though, Dan. So both sides own some of this, right? So where do CTOs and investors talk past each other? So Kirby mentioned stable, right?

6:41If I said to Devin, this platform is stable, I'm not sure he would know what the heck I was – I know in my head what I mean. The lights are on. The lights are on. It's working. And so there's two parts of that, Dan. And the other part of that is that the 90 % thing, right? Part of my argument about the 90 % thing is, you know, running engineering teams, my bosses, the managers would always talk me into, yeah, but it's done, right? I'm like, no, no, it's not done. I showed you this thing, but we still have this piece to do. So part of that, you know, is this talking past each other. So do you see that?

7:13Walk me through, like, what are the big phrases that, you know, the investors say, the CTO head spins and vice versa in your experience? I think first of all there's a fundamental disconnect between what done means. Software is never done right so software is a you know almost like a living thing that you work on and I think one of the key things we assess when we do a DD is also Howard's view is perfection is an anti-pattern in technology just doesn't exist and it's a you know BS thing to even you know discuss right it needs to get the job done and it needs to enable you to continue iterating on it right so architecture needs to be malleable, right?

7:52It needs to be something that can evolve over time. So I think ultimately the big topics we look at is trying to understand, are we talking to a team that is able and willing to commit to making decisions and learning from such decisions, right? And having the data to actually be able to learn from any decisions they made in the past, right? So I think this is something that's both difficult and fun. It's actually the funnest thing about engineering, right? It's, you know, you get paid to learn, you get paid to be in this engineering playground and sandpit and, you know, do fun things now with the help of agents also.

8:29So that's also fun. I want to ask, I split a question here because you mentioned a phrase that I like, right? So vibe coding versus architecture, right? And it's a little off topic here, but a lot of private AI's will, they hear vibe coding, right? Software is never done, which is great for playing, but that's not architecture, right? So talk to me a little bit about before we dive into the difference between those two from your viewpoint? Yeah. So if we think about just like overall in software time, it used to be, hey, we've got this team. How can we launch software faster? And it would be like, okay, can we hire more engineers?

9:01Oh, there's going to be a constraint right there. What's the constraint? With AI, you are able to build software faster. Our mental model has shifted. Okay, what is stability? And can it scale? Because in our plan, we want to double the size of this company. And we probably have some other metrics that matter like time it takes for a new customer to go live and inside of this this is really where we see the most important measurements and if people are missing each other and so when i think a little bit about you know what dan was talking about the best tech leaders are the ones who have given up trying to explain how tech works they've done a code switch and they speak in terms of like say for example their financial counterpart so whenever we're on the buy side and part of the value creation is like hire new tech leader it's because the current tech team can't explain anything.

9:45They're spending money, they're doing stuff, they're using tech terms, but overall the senior leadership and or the investor cannot know, okay, what you're doing right here, is it increasing revenue? Is it saving us time? And what's the ROI of it? And if you can't answer those foundational questions, that's where you see like a big leadership shift on who's doing it well versus not. Well, let's talk about that for a second. That's interesting in terms of whether, as we said earlier here, whether the senior management team is even involving the CTO in the business discussions, right? Yeah. To know whether what's the ROI of this feature or what's the ROI of this platform, right?

10:18But it goes back to the fundamentals around management. So, you know, this isn't a new problem, right? We've been writing software for decades, since the 70s. First, it was lines of code, right? Then it was, you know, Atkinson's famous 2000, right? Minus 2000 line, which is really clever, right? There's a line usually pinned on Bill Gates, something to the effect of, you know, measuring lines of code is like measuring the weight of an airplane to judge its efficiency, which is kind of stupid. He may never have said it, but it sounds like something the old Bill Gates would have said. Sounds clever.

10:47There we go. Yeah. Sounds clever. Good old Bill. Story points. So the guy invented it as even sort of said, hey, this may not have been a great idea. So look, now it's token counts probably, or at least that's what people are saying. So same idea, bigger cloud bill really is what the issue is, right? So, you know, Dan, lines of code, story points now tokens why do you think these things have never worked i mean i think you said it yourself with the example right in the end the best code is sometimes taken or oftentimes taking code away right refactoring and making something more efficient more effective um you know upgrading maintaining a code which oftentimes means you know you write you delete lines of code so if you look at the metric of lines of code that's just you know a bs metric and and also not a good you know proxy for progress.

11:33That being said, I think it's kind of changing in a way now that engineering and commercial realities are more connected and intertwined than ever, token cost, right? I think there is a reality in the end, companies need to act somewhat profitably, right? So that where you don't want to burn through tokens mindlessly and you need to have a strategy, you need to have a good harness, you need to, you know, design context, you need to make sure you don't just vibe code things, but actually iterate on your architecture, right? In a strategic system-driven kind of way. And that's oftentimes a cost question.

12:10So I think I would say many classic metrics, such as lines of code, are total BS and not worth it. They're fun to look at, but not worth it. And maybe they even might help you approximate the size of an application in a way, right? It doesn't mean the size of customer value or how many libraries, frameworks, and whatever boilerplate stuff you have inside. But I think the tides are turning a little bit that now there's a commercial reality that exceeds human cost very often, which are the cost of salaries and so forth, and is a very variable cost, a more complex problem costs more tokens, and maybe the outcome is not worth it.

12:51So you need to manage this very effectively. So we see good companies work in tandem now where the engineering teams and, you know, CFO and their colleagues collaborate together to make sure progress is something that can be done in a commercially attractive and feasible way. Yeah, I would also say that, you know, if there was a triangle and you've got investor, let's say CEO and CTO, sometimes inside that space, what the question the investor is asking, let's say board level, is the most sobering question then to the CTO. We were working on a project two weeks ago and the CTO is like, hey, my CTO, my CEO read about a thing called a dark factory.

13:29A dark factory, for those who don't know, is basically vibe codes being written and you don't really know if it works or not. The robot said it was tested, said it works. And to an investor, they probably hear that. They're like, okay, that's very interesting. But if I'm going to hold this company for five years, what if after two years it breaks and we can't serve the customers and we're just kind of stuck here, trying to fix this dark factory so there is a little bit of a relationship on are we all on the same page with our goals how we're measuring them and then trying to survive a little bit of the hype in the news cycle back to stability point right what's the hottest factory what's the hottest thing in ai like you know someone who's really taking the red pill they're like all right cool i want a thousand agents for it's like okay well who's now kind of checking that and then what's the read like what are they doing is it like a good outcome i think that's a really good CTOs ask that though versus like, I would say, I would have, I would bet investors are pushing, hey, can't we just have these dock factories that it's all a magic box and it works as opposed to CTOs saying, guys, hold on.

14:30I think that there's a little empathy for our CTOs here. There are some where they're like, hey, they're not involved in the strategy. Maybe they're not aligned as much on the commercial measurements. There is a reality though, where every CTO is getting bombarded by every LinkedIn article, Wall Street headline, hey, we built a dark factory this week. Hey, can we fire half of our engineering? And overall, they're still accountable to stability. This thing has to run. And if it breaks, they're up all week in fixing it, not the hypers who are asking the company to change that. So CTO empathy is a really important right now, just because they also have to keep up with the times to know what is the right thing to do with agentic AI.

15:07I like what you said regarding keeping up with the times. I think we still do not really know the total cost of ownership of AI-generated code, right? Yes, correct. 100%, and we won't. The verdict is still out. But I think ultimately that's okay, right? In the end, we're in this industry of constant change. Change a little bit quicker than it used to be in the past, I guess. And so it's the reality we have to deal with. Transparency helps, right? So dark factories where you just don't know what's happening, I think are probably not what you want to do. but ultimately in the end, you know, just as you keep it possible for an engineering team to iterate on software, I think you're, you're probably going to be fine.

15:46Yeah. And we can talk about what we're doing with Parker Gill as well, because I think it has a nice balance there. Oh yeah. We're going to talk about that boys. And in the meantime, I think, you know, Dark Factory sounds like a great Marvel movie, you know, maybe John Bernthal is in Dark Factory, right? It's perfect. All right. So, so Kirby, my follow up to that is, so a company shows up to diligence, right? So we've talked about this. Maybe the CTO isn't tied in with the CEO. Some of this comes to the surface often when private equity comes in, because it's a change, right? So a company shows up to diligence, you're meeting for the first time with a good looking engineering dashboard, right?

16:17So what's your first question? So looks great. What do you say to that CTO as they start walking you through it? Okay. So if it was 12 months ago, the question would be the same as it is today, but then also a little bit of a shift has happened. The first one is, is who cares? Who actually looks at this dashboard. And that could have been the same thing if you had a spreadsheet that you pulled up and it was like, hey, we've got this great metrics. I was like, do you make any decisions on them? Who looks at them? Who cares about them? And also how do they align with, say, for example, somebody who doesn't own them?

16:48Like, did you and the leadership team make this? That has been the same. One thing in Diligence that's pretty fun is you just look at the URLs. Is it running from their local? Because what we see now is when we go into a Diligence, the amount - You mean the local on the CTO's laptop is what you're saying? Yes. It's like the amount of things that were vibe coded the night before for the diligence. Like, of course we have this sexy dashboard that we use for all these decisions and these great documents. It's like, not true. And people in the last even three months have just been sharing cloud artifacts with us.

17:17And then we're like, none of this is real. You guys don't use this for decisions. And it's also not shared. So where you see the disconnect of, yes, we're measuring. You say, okay, what are you measuring? Doesn't matter. And who cares? That's the classic problem of, and we see this all the time in smaller software companies, right? They're scrambling to put all the diligence materials together. I mean, this is a long time ago now, but we had somebody like for the first time using source code control, like two days before we're supposed to come in and look at a deal, try to load up this - Initial commit.

17:49Yes, exactly. So I think you're right. Who is using this dashboard and are you really measuring engineering? or you're running engineering based on that dashboard. I would tell you in most cases, I have seen that that rarely makes it up to the board level or C-suite level in terms of those metrics. It's a dashboard engineering uses internally. It's got a huge disconnect in my experience, right? So given that's the case, the models got better. AI kills software. I mean, I don't think so. I think it actually is a - It kills bad software. Kills bad software, but otherwise it's literally like - It enables good software.

18:27It's tribles. The trouble with Tribbles, we're going to have a lot more software, which is both good and bad, right? Tech Diligence got pulled earlier in the deal. And I know we covered this in a previous episode with you guys, which I think is important. So, Dan, how the timing and the weight of Tech Diligence changed over the last year as relates to these kind of measurements? Like, what are they asking you for now that they didn't ask you for 12 months ago when they call you on? I think, as you said, we discussed this briefly. But for reference, back in the day, Tech.D. was a bit of a confirmatory exercise, right?

18:58You'd call us three weeks before signing. It's like, hey, boys, let's hope there's no skeletons in the closet. Let's take a look. Those were good times. However, now the times are even more interesting, whereas our work has been kind of ripped apart and shifted left, where we do a first phase, which is almost exclusively AI-driven. Before we move into cyber architecture or whatever, or beyond the baseline understanding of that, everybody needs to understand, will this thing be around in the next five years, right? or would this be sort of, you know, the next cloud feature, I guess? And so the most dramatic shift has been, it's been ripped apart, shifted left with a fairly specific focus on AI resilience.

19:36Can they play defense? And is there something magical here so that they can play offense? And only then we go a lot deeper. Now, would you say that's that, I agree with you, typically it was done at the end. And I mean, that's an embarrassment if you're buying software companies. So that's not how we did it most. But if you're - But is it really like ultimately technology is the enabling thing for commercial success, right? So it's great, but ultimately, if it's a healthy company, if it's growing, if there's retention, da-da-da-da-da, those are the factors that, and rightly so, make investors excited, right?

20:08Technology is the vessel. Technology is the foundation, maybe also the enable for value creation, but it's totally okay that you sort of, you know, preempt the commercial realities because that's what a company is measured against. Yeah, I would say that what's shifted for us too is, so even before LOI. When we come in to look at the AI, here's the things that investors care about. The first one is, is this company saying they're doing something special and they want to get paid extra for it. Is it true? Is it smoke or is it something that you guys can actually prove out? And we can do that pretty quickly.

20:38The second one is, is they say, hey, here's our thesis. We want to bet on this team. Are they capable? Have they already shown evidence that they can reduce the cost of engineering, ship more code faster by utilizing these modern tools? Basically, it's like a university have they proven the ability to learn because things will continue to shift give us that assessment there and that's pre-LOI after that we get more into all the things that Dan had said and then after the close it talks about okay well we've got a baseline here how is it going to go up and to the right or down to the left with what we know from the company what I actually find really cool is what you said before regarding um you know cloud artifacts and and or local host stuff.

21:17I actually think that's a cool trend. It shows that people go back to experimenting. And I think in our diligence, most of the time, people are very honest with us and they show us, look, this may not be live yet, but let me show you the cool shit I just built. And we will be rolling this out. So we just look for potential, right? And we look for willingness to adapt and improve and change. And I think, so to me, I think these are super exciting times. And I love the fact that people are honest and don't, you know, deploy a vibe coded thing somewhere, but just share the cloud artifact with us.

21:54And then we kind of iterate on it. And we talk, we discuss. I think these are super fun conversations. Yeah, but it's also important. We were working on a deal three months ago, and they showed us this beautiful next generation platform that was all vibe coded and had no reality. And we said to the investors, they've got a great vision, but do not pay them a multiple for that thing. Pay them for the software they have today. And also, we need to prove out that they can maybe get to that future. Sure. And that usually, boys, that usually in today's world is that's a scalability and reliability issue, not a feature issue.

22:25Right? Features are cheap now. It's amazing. Yeah. Totally agree. And that's the problem with vibe coding to prove that we sort of got it right as I push this in front of my users. Hey, did we get this right? Right. But it doesn't mean it's going to scale. right? That's the whole challenge behind this. And I think this gets us back to this, you know, I completely agree with this whole vibe coding is amazing. And the teams that are experimenting are probably the teams that are going to make it past your assessment, right? As opposed to teams got their head in the sand. I'm a great example of if you're a manufacturing fan, which I started out manufacturing, little's long, right?

22:56Which is people think they start more things, they get more things through. That's wrong. There's a mathematical model of how much whip you should have, right? Right. I am a great example of AI has allowed me to create way more projects, but my finished projects haven't increased, right? True. Now, that's a discipline problem for me and a discipline problem probably for teams. Or a prioritization capability. Yes, in my case, yes, that's absolutely true. Let's be honest with me. It's a discipline problem. We've seen a lot of projects, but we know that you're happy with projects. But that's okay.

23:24That's okay, right? Because now I can, with Vibe Coding, I look at things that I built and threw away, right? And I wouldn't have even tried to build it three years ago or two years ago because it would have been too much work for me to do it. But now throwing it away is the right idea. I built something like, we shouldn't build this. We should buy this. Or this was a dead end. Those things are okay coming out of VibeCoding. remember how you know maybe in 2010 12 whenever you know envision the first like prototype uh you know good prototype type applications came out where you can like link together back in the day photoshop templates right this was way before sketch way before figma and all these uh tools i love that it was so cool because in the end you can actually test things with actual users right you can design something very quickly and back in the day coding was hard right right and and And you can connect, have these, I look, click things, and then it would load the next page.

24:21It would feel like an interactive prototype, right? And vibe coding is that very often, right? It's just an actual... Right. Exactly, right? So in this one, when you mentioned about the factory, here are the big shifts that has happened that allow us to measure more. So if you think about people, process, and technology, learning and building is evolution. And so we say, hey, we can look across the data to see how many people are actually building out, learning, throwing things away. This is showing that they're exercising new muscles in the modern economy. Great, but that's only one pillar. The second piece you talked about about the factory.

24:55Thankfully, this is a new shift now, is that if you are doing agentic coding, you have to put your entire process on a workflow. And where things maybe that weren't like that. You should, Kirby. You should. Turns out you don't have to, but you should. But when you are actually exercising like an agentic harness and you're moving things through, we were always tracking things in JIRA, doing those terrible story points, trying to understand what throughput was. But if speed is now much faster, we're watching these things go through the factory and everything's a lot more measurable. And in the end, the last piece, technology, we're now able to measure the stability of that technology based on the process that we have online now.

25:34And so shifts recently have allowed for all of that data to be available for then non-technical people to truly understand what's going on in the factory. Because in the beginning, it used to be like, oh, they're down there building a machine, but it's also kind of part artistry and it's really hard to estimate. Now we have a lot more availability of quality metrics. I think it's a really good point. We move beyond, if done properly, if your agentic engineering workflow and the harness you build around it is done good, then you move beyond hero engineering. There's less key person risk, and then things you learn are written into instruction files.

26:14So suddenly, not just one person learns something, and the next person messes up again, but the whole system gets smarter. so specific errors and specific NR patterns are not done over and over again. So I think if done properly, that is an art. And I want to talk about that in a minute because that's that mythical 10 times engineer. But you made an interesting point with vibe coding and putting it out there. Let me just ask you the following question. Both of you can answer this. A lot of engineering teams have rallied around Dora as a way to say something we released is stable. Dora being Google's, well, Gene Kim and company, it's probably the most formal research around releasing software.

26:53So are you seeing that as a valid metric in terms of measuring success in terms of the scalability, in terms of reliability, scalability? I would say partial. So speed and stability are what Dora is based on. And we say, hey, speed's a lot faster. You can look at the factory. You can see how it's built. Maybe speed is less important now. And stability is the things that are out there that people care about. they're running they're working and the lights are staying on the thing that we're seeing is the best metric to combine now with Dora is the outcomes are you building features that matter and people use and there was this kind of hubbub about how oh is product management dead is you know what's going on here and what you mentioned being able to build out prototypes and build features that matter it used to be hard to put telemetry in to know the impact of what was happening now Now the best engineering teams inside of an R &D org is you have engineers, great stability.

27:51We can measure it. And then you have product engineering working together to then measure, are we going to reduce cost to serve? Are we increasing revenue? And to an investor, that's something that's better understood. The system is stable. It can scale to our financial model and things are being built that are driving something that I understand in my Patagonia vest, which is, okay, costs are going down, money's going up. I think that's an idealized view. And I think it's awesome. but I think also in the reality of private equity very often, right? The beauty of Door is it proves, if done correctly, that you can deploy pretty much any time, right?

28:27And it shows you that you're improving in terms of velocity and in terms of reliability, so you can ship things all the time. That's what everybody aspires, and for startups, that's the status quo. Realistically, in private equity, we still see desktop apps, right? We still see Delphi. We still see fat clients where business layer, so logic and presentation UI are intertwined and ripping that apart would be pretty much a rewrite, right? So I think realistically, Dora is a metric that works for some companies but not for every company. And the goal should be, the ultimate goal should be I can deploy whenever.

29:03So if the process is in place that I can ship code to my customers with decent reliability and stability whenever I want to, which, however, doesn't mean I deploy 250 times a day or so, right? So I think there's a degree of maturity-related realism, I guess, right? Well, that was the disconnect, right? The disconnect, I think, on Dora was between what developers see and what the business sees, right? It sort of started out as that, hey, I run a website, and I can release 50 times a day. Fine, you're running a website. That's not an ERP application, right? Right, correct. The ERP guy said, we only release once a year, right?

29:42and the customers are like, oh God, here comes the annual upgrade, right? Exactly. This is the idea of... Sometimes customers are even like, this is too quick. Like if you, you know, when we look at pharmaceutical, software that serves pharmaceutical companies, for example, you see protected branches because there's so much customization and they're sometimes like seven to 10 years behind and that's terrible, right? On the other hand, it's the reality of an industry that's so massively regulated that even a minor update means weeks and weeks of compliance work. Yes. But, but, but even there, I think part of it is we're in a world now where you should be able, even at that large scale applications to release more often, even if you don't or release in a way that your customers don't, your feature flagging everything.

30:26Right. Customers don't know it changed. Right. So it allows what I like to call the Dunkirk retreat. Right. Like you're slowly, you know, uh, holding the line while the brick boys get rescued. You know, uh, we're treating as opposed to just abandoning. yeah there's there's also a reality and you mentioned it earlier about say for example big rewrites what's happening here kind of what's your gut feel on where you guys are at in the migration and with ai now especially for non-technical like say the finance team that wants to get a true update you can deploy a version that's not in you know a different production environment you can run simulated tests against it you can prove that even though you're going to do a yearly release over here the software is going to work and it's not going to be like that you said like the prayer that eventually hopefully this thing goes live and it's good so i think ai has also helped even some of the legacy processes get more um transparency and accountability to then say for example the board like where are you at with this thing but i think the other thing to to to backfill on that i think a little bit kirby is the you know ai analysis of common languages you know popular languages like python or something is great yeah if i'm going back to PowerBuilder or Delphi, we're not going to have the same level of agentic expertise in pulling out those languages.

31:43Again, it doesn't matter. But that's for my investor partners. If you've got something written in Python, great. I can look at that. If you've got something written in Visual Basic, we may have a longer update process, if you will. We talked a lot about the engineering dashboards themselves. So let's do this. What does an investor need that those don't show? Like when the investor says to you, we need to fix engineering. What are the things they're asking for in English that you think, yeah, that makes sense, that engineering isn't focused on or doesn't use? So I think when it comes to like, what does an investor need?

32:22The best place to start is what do they understand already? Headcount and costs. So, hey, I have this amount of expense, what am I getting for it? And with this one, you know, you're able, you can break that down very simply to say, okay, we're releasing this amount of code, which we say is a very good. So we say, hey, we're going to shift that metric to with this amount of people, one, stability, the system is stable to take on the scale, whatever the sales team is selling. And then from here, how are we producing more features in our overall end-to-end workflow? And what they want to know is saying, hey, we're able to release more features that have an outcome and the cost of it's going down.

33:00Which with AI, there are some things that are true. You don't have to go full dark factory or just agents write all the code. The best answer is somewhere in the middle, where for all of your kind of junior engineer tasks, I want to see agentic coding is being used. For all the things that truly matter, architecture, I want the humans in the loop, making sure that they're defining what those are. And I want quality to be the same. Quality is always going to be running this or quality is improving, which we're measuring that with like test cases passed or things like that. So in this area where an investor, what do they want to know?

33:31I want to know that my costs are going down relative to what's coming out of the system. Or, I mean, in some cases, I don't know this enough costs are going down. If we're keeping costs flat and productivity is going up, we're selling more again, also fine. But that's a great point that oftentimes a CTO has no idea that that's what the business is trying to ask you to do. Which one are you trying to do, right? Like, are we trying to grow revenue, keep costs flat? Or we know we can't keep costs flat, so it's got to go up a little bit, but a slower ratio than revenue, right? That's where it gets complex, in my opinion.

34:04Two things can be true also, right? As a PE, I care about risk mitigation, revenue growth, efficiency enhancements, and then multiple arbitrage, right? And the different engineering buckets these things can operate in, right? And sometimes it means, you know, maybe the team is too big. Sometimes it means the team is not big enough. Sometimes it means throughput is not quick enough. Sometimes it means that test cases are great because there's too many bugs that make it to production, etc. So I think it really depends on the sector, on the majority of the company and other things. Yeah, and Dan just lists out four things.

34:41What's been the big shift that we've seen and why the work has shifted less is the fifth thing is they care about what is AI's impact on this business? Are we playing defense? Are we playing offense? Can I sell this company? after the hold period, and how is what we're doing with AI going to impact the valuation? And so when AI is part of the strategy, or they're doing something in ops, they're doing something in engineering, they're building something proprietary, those things are all measurable. And if I'm an investor, I'm going to say, hey, when I exit this company, everything should be going up and to the right for the things that create value.

35:12And that's really in that exit readiness that we had talked about in the previous episode. So that's probably the perfect point, the perfect transition. So perfectly done. So So we spent the summer, or Devin spent the summer with you guys, building out an audit for the lower middle market, right? And the CTO, me, stayed out of it on purpose, right? Because it's a super interesting way. I want to be an observer of the process without affecting it. So talk to me about the audit. Walk me through the broad categories of your audit. Where are you pulling the data from, and what are you trying to get out of it?

35:41Yeah, so for the audit, and especially for the types of companies that Parker Gale's working with, they have something they're doing that makes them special. They also are working with Parker Gale to get to like, what's their next stage? Where are they at? So the audit focuses on two key areas. The first one is AI engineering. Where are they at in this process? So you've got the engineering team. They're building certain features. Maybe they're training a model. They're doing something special. And in it, how much are they leveraging the technology responsibly? And so it's not overbuilding. We're not looking at token costs.

36:11But more so using some of those classic measurements around speed and stability. and how much are they taking these new, very talented interns, AI, to help them build code faster so that in their next chapter with Parker Gale, they're not having to triple the engineering size, but more so grow in the right way and get everything on a workflow so it can be measured. So AI engineering is really where we put the first pot. The second one is around the product layer. This one's a little squishier because what we see on the buy side and sell side is that over the last year, a lot of worthless AI features have been built.

36:45A lot of chatbots are out there that nobody uses. Are you talking about Microsoft Co-Pilot? Let's just say it. Co-Pilot. And so in doing so, when we're looking at these companies, especially with the Parker Gill companies, we say, okay, whatever you're doing outside of engineering, you are spending. That's a cost measure there. So what are you trying to get from an outcome perspective? And working with the CTO to bridge that gap that you had talked about on make something that the spreadsheet team can understand and what that looks like is at the product layer are you offering some sort of feature with ai that you understand the cost of what are the unit economics of that feature and in this next chapter of the whole period if this company triples do you understand cost there the second area and probably more effective is if operations is running on a workflow let's say from implementation all the way to renewal is that on a workflow where you're using AI to help reduce the cost to serve for operations.

37:40And this is a space where it's very measurable. Everything's on a system. And as part of what we're working on with Parker Gale, we say, okay, monthly reporting on these key things, as well as the context. What is the initiative that are currently working on to get to the next stage, either engineering or say, for example, in optimizing operations? Walk me through some other specific AI things, right? Because for example, You said training a model, right? Yeah. That to me is a giant red flag. If a portfolio company says to me, we're training a model, I'm going to be, ooh, ooh. Let's say they're fine-tuning the model.

38:11Have we tried RAG first? Have we tried MCP? So are you trying to pull those things in too to say, look, these are frameworks that have become common in AI, right? And you should have those frameworks too, right? Like, should you be put? MCP probably is one of those weird things that crosses between engineering and product, right? Engineering has no interest in doing this because they see all the risks, right? product says everybody wants an MCP server. There's a classic challenge of what do we do, right? So when we take those two items, if you're fine-tuning a model and or making an MCP, those all go in the product layer.

38:42Every company in the Parker Gale portfolio is different. Have some that makes them special. But inside of this, we say, okay, great. You built a thing. You built an MCP. Is anybody using it? Is it driving an outcome? And then if they're like, this thing's amazing, everybody loves it. It's like, do you understand the cost? because as we're looking at it monthly, if the costs are going up, it's like, okay, well, maybe we need to pivot this off of a frontier model to an open weight model so we can get a better understanding of the unit economics of it. But by looking at that key area, product layer, what's happening, and then the cost of it with Parker Gale, we're able to better understand that because at the board level, it's not their fault.

39:19They're reading a bunch of headlines. They're like, oh my gosh, Uber just spent this amount on tokens in the first quarter. What are we doing on tokens with our AI engineering? Yeah, but that's - Oh, look at this. It's actually in a board meeting. We should be throwing our coffee in the board's face, right? Because we're not like - Not Parker Gill, but others. But what we see is basically is like, as the industry is moving so fast, there's things that people get excited about. There's things that people get scared about. And the reality is somewhere in the middle. So when you're doing monthly monitoring, it's okay, what truly matters?

39:49Are we being more efficient, speed and stability in engineering? And then at the product layer, are we building stuff that matters and we know how much it costs? So you bring up a great point. So let's go back to the open weight model for a second because this is a hot button with me when you talk about planning. So somebody builds an MCP server and they're going to use the latest Claude model, whatever. They've done no benchmark testing to see what those tokens cost. The fact that they now say, hey, we've got to move this to open weight means, yeah, you should have been fired as the CTO or the product guy.

40:17Like if you knew that a zillion of your customers needed to use this and you knew going into it that you need to be able to scale it without expensive tokens into the situation, totally fine to test it on a frontier model. It's the easiest, simplest thing to do, right? But that gets into this, did they go the wrong way in terms of, you know, that's the only thing they understood was a frontier model. So that's what they did. I think that's a super important point. In the end, I just saw a statistic that about less than 30%, 28 % of tokens consumed are being consumed by frontier models now. The rest is open-weight models, right?

40:51So the trend is good. Which is good, right? But I think the read is for me that maybe 20 % or 30 % of the most complex requests get directed to frontier models. But 80 % or 75 % or so, different open-weight or smaller models are good enough, right? So I think a key capability for teams is actually now to be able to discern, you know, where to direct a request, right? And what to use when, right? So what you said before is, yes, it's easy to use a frontier model and it's beautiful because, you know, they have this really, you know, mature developer experience with strong APIs and developer tools and stuff.

41:26It makes it easy, right? But then again, there are many requests where something more simple is actually good enough, right? So a key capability that we probe when we look at companies, both on buy-side due diligence audits, sell-side, or portfolio monitoring, we try to discern, are they capable of making that choice consistently? And I think, does that show up in your framework? This, to me, is relatively new, right? I'm pleased by that statistic. It shocks me, actually, right? But I think it's the right way to go, right? both for mother earth we don't need the latest and greatest models despite the fact that elon must thinks we're going to put the data centers in space which we all know is insane but that's so are you looking at that like the team's maturity and trying to use non you know frontier models and non you know clawed and codex versus trying to build up their own you know underneath the cover servers routers oh you know ai routers for this stuff i mean the reality is so say for example example, we'll use the Parker Gale example.

42:23We do this for other portfolios. But in the Parker Gale portfolio, once you have a baseline, we've gone through, we've done the work to understand, okay, what are your usage metrics? What's your engineering metrics? Modified them based on what is the particular domain that company's working on. To Parker Gale, they need to see a simple view. What are all my companies? Where are they trending? And what's happening from the terms that I understand, us being efficient. For us in monitoring there, we say, okay, great. The industry will continue to change, but the metrics don't have to. For the features that you have, either in ops or product, are they turning in the right direction?

43:00Do you understand the costs? There will be something that will come out next month that will help optimize costs even more, but we need to be able to see what the costs are today. What are they next month? And then comes in the strategy. So why we partner with portfolios like Parker Gale is that we're able to then take the latest and greatest of what we see as the best and then help influence that. You mentioned MCP earlier. Talked about is there an open router they could do. But whatever new tools come out, we want to make sure across the portfolio, here are the best strategies. And then also, if something's turning the wrong way, you could see all your portfolio companies and you can see one that's kind of an outlier.

43:33You say, what's happening there? Which to me is part of the promise of actually supporting the companies where it's like, hey, I have more of a view of the market. I can help steer you in the direction in which it's a very fast-moving industry. So for us, monitoring is the most important. now at scale how often then do you do you check in how often you're going to check in with both the investors and the portfolio come in a case we have a baseline like that makes sense best is monthly there are something we do quarterly because it also aligns to board meetings but it's focused on two things the first one is capture and then context capture is they have all these systems today we already talked about how everything's on a workflow or it'll get there which is part of their you know first task but from their capture is either simple exports or even utilizing connectors from Code & Co where we're getting those automated data feeds.

44:17After that context, there's stuff going on in the industry. There's stuff going on inside of their department. We want to make sure that we're not reporting any story up and to the right or down to the left that doesn't actually have the context of what they're doing. So what we've accomplished is we're able to translate to a non-technical person why the numbers are meaningful, but then still giving the technical leads the ability to talk about their craft and what they're doing and what they're trying proving that they're still credible and also like a person that can drive the overall strategy and and how do you get engineering and product teams on board with this right uh the first challenge in my opinion right the first one is you got to be real so whenever we come into an organization they've they're already getting bombarded by everybody about oh you should do this that give me this number do that so at code and co shocking gerby at code and code our first thing is that we're all technologists like we still build you have to do product strategy, you still do engineering.

45:11And when we work with the teams, our first thing is we just take the pressure down and we say, we are not here for gotchas. We are simply here to help tell your story and hopefully translate in a way where you're already having to do this for everybody, but we're helping you one do the monitoring, but then also the framing to a non-technical investor. And we do this because we do this all day at Code & Co. If you're actually being helpful to a product engineering team, they lean in because they're like, hey, that's more arms and legs, hands and hearts. That works. If you're not being helpful, they'll kick you right out, especially if you don't even know what you're talking about.

45:41So that's kind of one where you have to be credible to get their trust and you have to be truly genuine that you're just there to try to monitor and not surveil. So surveil is the evilest version of that, right? The non-evil version is, okay, you guys set up a framework and now we just game the system, right? Because once something's being measured, right? And again, this was always, this is the classic story points problem, right? Like, yeah, my engineering teams, they always, you know, they overestimated the engineering points for, you know, we got to change the font on this one screen, 80 points.

46:14Okay. Right. Like the game in the system. So how do you keep them from gaming the system in this, in this world? I think, I think one thing, maybe before you answer that, one thing that's really important to, to, to point out here is we are here to help, right? Back to the question that you had regarding, how do we get them to actually do this, right? Because we understand they have lots of things on their plate, right? But in the end, we're here to help. And we have the benefit of working with brilliant people, you know, all day, every day. And we do this globally. And in the group, we're going to do probably like 300 deals or so this year.

46:46And so we know what good looks like. And in a day and age of AI, change is so rapid and so insanely fast and so disruptive. I hate that word, but I think it's somewhat appropriate here. It's a game changer. Oh, God, you did it on purpose. Yeah, exactly. It is my setup curve. It's my least favorite. It's game changing. It's game changing. And so we kind of have an understanding of what good looks like. And what I find is it's a very constructive thing. It's not about surveillance. It's about helping people embrace the weird and wonderful world of AI and cyber, both of which are moving targets. And both of which are also increasingly intertwined.

47:22I mean, you read the news, right? So you see what's happening out there, that AI escapes their guardrails and do whatever it wants to. So I think these things are relevant and you need to be. Did it escape? Because I think in the case of Hugging Face, I think it was pretty much an attack, my friend. Well, yeah. But you're right. You're right. It is an escape. Never waste a good story. No, but it is an escape. And AI is non-deterministic. Exactly. So I think it's just important to understand, at least I have never seen a time where changes were so drastic. We're super grateful that teams actually enjoy working with us.

48:03And we're here to help. We had this case just a couple weeks ago in an actual sales side deal where we found they didn't have an MCP. And it was actually quite surprising given what the company does. And one of our guys actually spent the weekend building MCP server for them. And so we want to be hands-on. We are in their team, whether it's portfolio monitoring or due diligence. We want to be respectful in terms of they've built something great. Otherwise, you, Jim, would not be talking to them. As an investor, you look at great companies. Great to us is always, again, in lower middle market, great to us has always been the product.

48:40right you may do an aqua hire in the west coast you know you're just saying hey it's a great engineering team we're just gonna we hate their product we're gonna buy them because they're amazing guys right we don't see that you know we generally start with it's a pretty good application which means it's sticky with the customers you know that they've been on a long time customers don't want to jump off it right so we've already got a little bit of a you know uh sort of a base there which is somewhat different it's you know now it's the most difficult thing right building a product that people are willing to pay money for power to you awesome right the rest is foundation and can be changed but a good product experience something that solves a customer problem that's what you measured against right yes i think i think also if you want to game the system you know we used to be pretty hyped like those token leaderboards on who could spend the most tokens but we pick metrics that measure that matter that are hard to game which is like time money efficiency and when you do capture you can game capture but you can't gain context why did that have the variance and from there anyone who's reading it you know a board an investor a senior leader is not technical they still have a bs meter and they're like i don't know i don't believe that game and when we go into work with an engineering team or an r d org we help them understand that this is pre-exit readiness if you have good telemetry in your house the plumbing is good the foundation is good then you're able to basically set up this exit with proof that you are doing the right things and the metrics are going the right way isn't it hilarious that only a couple months ago people had actual token spent leaderboards yeah it's it's insane hilarious i'm the most wasteful but that's a that's a gamed metric you know right but but the other thing that again uh you know me silly rabbit segues are for kids right it brings to mind the little's law problem though of you starting too many projects, right?

50:32One of the things that AI lets us do is start a gazillion projects that we don't necessarily have to finish. So that goes back to that whole, the token maxing thing, right? If you've got some engineer having 10 ,000 agents do a bunch of stuff, where is the human in the loop checking any of it? Now, AI can check itself. I get all that, but there's some point you get so many threads running. I don't know how a person can manage that, right? That becomes a problem with doing too much of this token maxing. Human must be the accountability and judgment layer, right? Humans must make the difficult calls.

51:04And that's strategy. It's working within constraints and hopefully making the right call. And data helps here. And this is why we're so excited about partnering with companies and helping them on that journey. We can help them make hopefully better decisions. So to wrap that up, let me ask you a question for both of you, right? Yes, sir. You've done an assessment. Sir, as my father would say, I work for a living. you know yes sir mr sir my dad was a yeah you will deploy for breakfast at oh 600 hours kind of guy you know push-ups yeah yeah my dad was navy and ibm and i was a 70s hippie you can imagine how that went over you know short work at coding go uh there you go neat pictures punk rock and pe what what's what should so let's say you did an assessment today right we closed the deal like and forget what's the specifics on it what should the investors get out of it over the course of a year.

51:53And this is more of the, not, oh, they gained 10 % or whatever. So this is new for a lot of people, right? Being able to understand what engineering is doing and have a play in this process. So over a year, what do you think happens between the investor and the teams if this is done right in terms of understanding this and measuring it? I would say that one of my tells is, is that value creation, a piece of value creation playing a piece of junk that you threw away, or did you actually care about it. And when we work with other private equity firms in that first year, the things that are in the value creation plan, either a key hire, set up agent encoding, there is an accountability, maybe a quarterly check-in as something where we're measuring towards zero to one.

52:33And then once we get to zero one, what are we tracking from there? That to me is one of the biggest areas of like, was this work for waste or do they actually care about these things in the hold period? You know, in a DD, we learn a lot, right? But we also have to jump to conclusions because you've limited access, time, access, and all these things, right? So as part of our work, we shape a value creation roadmap. As part of our work, we want to go deeper. So after DD, we want to do like an inside-out DD where we go a lot deeper and include the team where possible, right? And we connect to systems and get more data.

53:07And this will help us inform and also challenge us, you know, because we do mistakes too, right? So we want to make sure that the recommendations we give are the right ones. And I think this is where it's so, you know, beneficial for both sides involved. And this is how we segue and transition into a value creation support phase, right? Based on the educated guess, right? That's part of diligence within a constraint that a DD process has to a more inside out diligence with a lot more data access to, you know, hopefully real-time data feeds or almost real-time in terms of data export data feeds that help us together collaboratively decide decide on and execute roadmaps?

53:47We've only worked on one project this year that did not have a material change in the value creation plan for product and technology. Material changes, key hire, do a key system, execute a key migration, whatever these things are. And I only know of one deal where that's just a perfect company. And so because they're mostly imperfect in this area, those things still are true and you have to put accountability to it in a true plan. And say, for example, you work with a firm And they're like, we do these QBRs and we have no idea what they're talking about on the product and technology roadmap, nor if they did that whole honey-do list that we gave them after the close that was a big Excel sheet.

54:25And so that's a little bit of the disconnect. So let me wrap it up here and see if you boys agree or disagree, right? So three things to take back to your next board meeting. One, you can't rank individual engineers without breaking what you're measuring. Would you agree with that? Disagree. Okay, good. Disagree, yeah. I mean, with Code & Co. and even with Parker Gale, we can see what engineers are contributing. We can also see how they're contributing. And so ranking is just a good way to understand basically a sort order. And if someone's at the bottom of that order, it's worth a conversation.

55:01Bottom in what way? Explain bottom. It could be bottom like not using AI. Like why is Jim not using AI? Or why is Jim using all AI? But there's a conversation that needs to be had about your engineering org. and frankly you have the data to have an informed conversation good better ugly well so let me ask it let me answer another way the board sees that and they see jim at the bottom of this list now i've spent 90 of my time on the whiteboard team doing some architecture 90 of my time with the product team this quarter because they had product questions on engineering questions so i i showed up terrible on your dashboard right yeah but i did the right thing here right into us, from a board level perspective, they need to see things outside the engineering context.

55:43But I'm saying that if there is any accountability or a space in that, I think that's interesting just to see what engineering activity is. If you can just understand like, oh, that's Jim. We love Jim. He does great architecture. Or he's out there talking to customers all day. These are things that probably have never been said, but a different Jim. They never said anything about me. I can promise you. I mean, we're also in the business of relationships and humans, right? Ultimately, this is what conversations are for. And we don't want to be just looking at data feeds and look at data feeds. They might paint the wrong picture.

56:14In the end, human, accountability, judgment. And this is where conversations come in. This is the human in the loop bit. Right. But I guess that's my point with just measuring what an engineer produced doesn't give you a real... 100%. If you look at the Code & Code repository, you'd be like, who's this Kirby? This guy must be an amazing engineer. No. It's just a bunch of work. Yeah. But it's okay. you know yeah like i go by points i know my place i know my place all right so two would you agree that you need to pair any speed number increase in delivery more products with scalability and stability 100 yes okay that's good one for one here we go this is the last one here i decide whether i win or lose okay so three i think what i got out of this was measure to learn not to punish correct is that a is that a key thing right nice and once the number decides somebody's pay right then it stops telling the truth, in my opinion.

57:10I think that overall the incentives are important. So you measure what matters. And that's where we see in the best engineering organizations, they know what they're doing is aligned to the bigger strategy. And the bigger strategy is going to lead up to an exit. But overall, engineers and product people hate building stuff that nobody uses. Of course. And also inside of how they're booting their craft, There's artistry there. And so all these new paints and tools are coming out to do your agentic engineering. To me, these things can be aligned to bigger incentives if they're the correct incentives with the investor team.

57:46What Kirby said. Wow, that's an Ibbid. We got a ditto. Is that from Ghost? Ditto. That's a ditto. What did I miss, boys? Did I miss anything? No, I really like what you said regarding measure to learn, not to punish. I think that's ultimately the goal of collaboration, right? You want to make sure that you improve output within what's possible, what's appropriate, and what should be done. And not just data for the sake of data. I think that's a really good statement. We're going to steal that. Yeah, and I would also say that. You steal everything. I don't copyright anything, guys. I would say that if you want to find measurements that are meaningful to the investor team, work with the investors on them and then work backwards from that.

58:28in all the portfolios we work with for monitoring. If you ask all the engineers to get into a room and find the best metrics, they will just fight about it. And so it's starting with the end in mind and working backwards from that. We find that when you do that, then engineering leadership understands, okay, yes, I can simplify this, but then also make sure that it's material in what we're doing and reflecting the investments. That's a great point. All right, gentlemen. So for more in Quote & Co's work, there's a link to their site. They're all over our portfolio, So there's no way you can't find them everywhere.

58:58But there's a link to it in the show notes. Kirby is the contact for U.S. firms. Thanks for having me. Like and subscribe so you never miss an episode. Punch that. And check out for now, gentlemen. Thanks for having us, Jim. Thank you.

59:12Thank you.

59:26Pour yourself a drink and have a seat.