When Will the Models be Good Enough?

0:07Aaron: Good morning, good evening, and welcome back to the Enterprise AI Show. This is your host, Aaron, and today we have the third in our Models and Markets series. Brian and I asked the question, when will the models be good enough? We explore why now with an example from the past, cover the current state, and finally dig into a possible future state. And all of that is coming up right after this break.

0:32Today's show is sponsored by ShareGate. You're out of time. Copilot needs to be deployed ASAP, but your tenant really isn't ready for it. Years of data, permissions, and users lurk in its shadows, ready to be exposed by AI. ShareGate Protect sees it all, so you can find the exposure risks, fix them fast, and deploy AI with confidence. Microsoft 365 Governance, they've got this. Learn more at sharegate.com slash protect.

1:00Today's show is sponsored by Nasuni. There's a growing gap in AI right now between what's possible in theory and what successfully works at scale inside an enterprise. The difference comes down to unstructured file data. Many AI initiatives struggle because the file data they depend on is scattered, unstructured, and disconnected from where and how work actually happens. Nasuni changes that. It brings your unstructured file data into a single, secure foundation so AI, both generative and agentic, can access it with the context, governance, and performance it needs in production. Bring AI to where your unstructured data lives.

1:35See what it takes to activate your data for AI and request a demo at nasuni.com slash AI.

1:44Aaron: And we're back. And hey, Brian, we're continuing this series. And you know what? I'm sure by show for, we will have a name and it'll be in the show title and it'll be amazing. But as we're recording them, we don't have the name. So we're going to continue our, our thought process here of kind of asking some questions and then like, okay, the idea is I'm going to ask a question. We've got some topics. We're going to talk about what is it now? Where are we in the past? You know, something we can compare it to, to kind of put an idea in everyone's head and allow everyone to think about it. And then let's do present and future state as well.

2:17Aaron: So what's today's topic, Brian? Well, I think today we're going to talk about something that feels like it is very, very open-ended, which is essentially, you know, at what point do we think either, you know, off-the-shelf, you know, AI technology is going to be good enough for your business without a whole lot of customization? Or do we think there's going to be sort of a tipping point where people say, either you just don't understand my business well enough, I will figure stuff out on my own, or, you know, kind of to the point of the now sort of infamous Alex Karp, you know, complaining on CNBC or rant on CNBC, you don't want to give your alpha to these companies, these model companies.

2:57And therefore, you know, off the shelf won't be an option for you because, you know, your concern about giving up your business alpha outweighs, you know, the value you get from just buying something off the shelf.

3:08Aaron: Yep. Yep. Agreed. And so here's the thing that I kind of want to dig into on this, because right now I would say there's really two big trends in the industry. And by the way, they're not super exclusive. They shouldn't be super exclusive, but almost everyone treats them as exclusive. And so first of all, there is the idea of model customization, taking an OSS model, not necessarily one of the models in the sky, but taking an OSS model and fine tuning it and doing some post-training on it and doing some other steps to basically take it and make it your own. And like the one I like to use the most is, okay, let's say you're a legal firm.

3:43Aaron: You want to train your model to be a lawyer and not just off the shelf. And so that's where model customization, that's certainly one aspect of it. And another one that we're certainly seeing now, because honestly, I think it's just so much easier is model harnesses. So a wrapper around the model to then a lot of times we talk about harnesses in the form of agentic and agentic tasks. But we're increasingly seeing like, okay, the harness controls it. It's a little bit of guardrails. It's a little bit of like keeps it on task. There's all these other things that are built into the harness. And I really think to kind of put my, my, my very limited data science hat on for a second, by the way, model customization, it's hard.

4:21Aaron: It's really, really hard. And so a lot of times it's easier to just build a harness around it. Like model customization is data scientists harnesses. A lot of times is more of a developer kind of function. And so who's doing it and what level of complexity is needed. And to get, and where I'm going with all of this though, is like, I want to compare it to the past. Here's where I see this going. Every enterprise has some kind of ERP or CRM system. And every single one of them that's out there, they need one. And oh, by the way, they're never the same. They're always super customized. You know, like I've been in many organizations, salesforce.com, SFDC, man, oh man.

5:01Aaron: And they're, everyone's a nightmare. And they, they're always working on them and they're always tweaking and tuning them. And it's just this living, growing, breathing thing that requires constant care and feeding. And I think the idea in the industry is AI will eventually be that, you know, to get really good AI, there needs to be my AI that is completely mine. It's not off the shelf. It knows my customers. It knows my process. and, but, but how do we get there? I think we're a long way away. And this is where FTEs, forward deployed engineers comes into play, right? Of like, why do we do that?

5:45Aaron: Well, because it's a bunch of people that come in, a bunch of consultants, maybe they're data scientists, maybe they're just programmers. So it's a combination of a bunch of them to do all of this. Yeah. Yeah. I mean, I think the analogy that jumps out and we've been trying to, you know, do analogy based stuff with these shows is, you know, the, the very first time I ever went to, I got invited or I had to go to an event that was the SAP event. I forget what it's called, but it's, you know, the AWS re-invent for SAP. And it was the very first time I went to an event in which 90 % of the people there were dressed in suits.

6:17And every big booth was a giant consulting company of some sort, a GSI of some sort. And it was just so weird and shocking because I was so used to going to, you know, whether it was a networking centric event or an open source event or a cloud event. It was hoodies and jeans and techies. And it kind of reiterated the point that that world, and this was all obviously CRM, ERP, HCFs, all that kind of stuff, is dictated by consultants. And I think we're starting to see the beginning of that again with these companies, even the AI companies being like, I'm going to have to spin up my own FTE company, which is, they don't want to say it, but it's essentially consultants who may be there for a short period of time or a long period of time.

7:02And the technology analogy that sort of jumps out to me is, you know, this feels very much like, well, you know, whether we called it a data lake or a data, you know, a data ocean or a data, you know, for all those different ones we talked about, you know, an iceberg and so on and so forth. It was this premise of, well, you know, I just want to take all my data. I don't want to sort it. I don't want to have to reclassify it. I don't want to have to clean it. I just want to dump it into this thing and it's just going to figure it out for me, which is, you know, essentially what you were saying is like tuning a model is really, really hard.

7:36And it's the same reason why, you know, people had such a hard time for so long, maybe up until when Snowflake came along, whatever magic they do, where they were like, God, I'm spending a ton of money on data lakes and data scientists and people that do ETL and all these sort of things. I'm not seeing the result of it. it feels like we're kind of going through that, you know, that, that sort of parallel path again. And, you know, again, because the people that are now doing gen AI weren't necessarily the data lake people, or maybe they were, they, they forgot about it. We're kind of repeating the sins of the past.

8:06Yeah.

8:07Aaron: And so I w I want to get your opinion on this because I actually, you know, I, I get to talk to some customers sometimes, but I feel like you talk to more customers than I do. what is your thoughts on okay one of the big things i think because i don't hear this analogy kicked around very much this is something you and i are kind of coming up with here but the idea here though is this because up until now the models and the harnesses and everything else like that is honestly everything is so sciency for lack of a better word of like you got a bunch of data scientists and a bunch of data scientists either don't work in the enterprise or, you know, you know, don't want to work in the enterprise.

8:48Aaron: They want to be engaged in science and they want to write research papers. And like, I know I'm being super stereotypical here, by the way, for everyone out there that wants to come out with pitch, pitchforks, but like, is this because like, it's actually a different industry that just doesn't realize yet what it takes. And so like, you're eventually going to have like the data scientists in the back room and all the guys in the suits in the front, you know, presenting at the trade shows? Like, where do you think? Well, I think to a certain extent, yeah, there's some of that, right? Like, there's a finite number of, like they say, there's a finite number of people that could actually build one of these models, let alone, you know, then go train them or whatever, you know, the adjustment stuff would be.

9:32I think there's a couple of things that we seem to sort of keep forgetting because, you know, I mean, right now, AI is the equivalent of like walking into a, an echoey room that is just playing nonstop ACDC all the time. Like it's just banging your head. It's just loud. You can't get it out of your head. And I think we forget the fact that like literally the entire, this entire part of the industry is four years old. The concept of harnesses is maybe, maybe a year old, right? And, you know, and people are like, oh, we haven't figured. And you're like, you know, historically, like, you know, you follow the sort of 10 ,000 hour rule.

10:12I mean, 10 ,000 hours is five years of experience of doing this stuff. But yeah, I mean, I think we've gone through these little phases so fast that not only do we kind of get frustrated by it, but the other part that we, you know, we seem to forget all the time, even though we sort of know it is, these aren't deterministic systems. But we keep trying to force them to be deterministic systems. We want the outputs to be deterministic. and you know so yeah that's why we start to have harnesses and people are figuring out harness engineering because it's like i'm trying to to cram a you know a round peg into a square hole and we're like why isn't this working that would be awesome for you know what i want to have happen so i you know i i think you're i think your statement and analogy is right i think there's certain things that we just we sort of forget or we you know we choose to selectively not remember these things.

11:02But yeah, I mean, I kind of keep coming back to this idea in the context of like, you know, should I be able to do stuff off the shelf or not? I just think there's such a missed opportunity, you know, especially with all these companies that are firing people and firing technical people. And you're like, the best thing you can do is marry a technical person and a business person together. The business person knows what they want. They know it as a back of a napkin, they know it as a whiteboard, and the technical person can go code it for you. And if all they're doing is building some harness level stuff for you, or some prompt engineering stuff, or whatever it might be, like not marrying those two things together is such a miss.

11:41And right now we're doing such dumb things within, you know, at the top level of like, getting rid of people that either know the domain really well, but we're like, oh, we're going to be able to run rough shot over them, or getting rid of software developers, because we're like, oh, they're not, They can't code as fast as these other things. It's like every one of your jobs is a software project, but they're not going to work all by themselves. The AI is not going to magically learn your business and magically write the code and then magically understand the people part of it. You have to marry those things together, and it's just like that's not happening.

12:16And then that's going to keep pushing this problem of like, I can't use off the shelf, but I can't make it work for my business. either. And there is a way to get there. We're just, we're ignoring the most obvious way to get there.

12:29Aaron: So, so let me ask you this, let's kind of move into future state for a little bit, because I have this idea and we've talked about it many, many times on the show before of like, okay, let's say for a second model customization today requires FDs and model harnesses requires developers and consultants and all these other things. And ERP and CRM systems, like you said, It was for once upon a time, super dominated by consultants. Honestly, I don't know if it still is. But does it move to a point like in my mind that the reverse S curve on the commodities of like everything's bespoke, everything's specialized, and eventually you get products and products leads to a bunch of traction and prices go down.

13:12Aaron: And this is what eventually leads to commodity. Is this market ever going to lead to commodity? Is there going to be products that are customization? And will the market continue to move up in S-curve? Or is this just something that's like, hey, we're just going to figure out different ways around it like we did with the ERP and CRM systems? I mean, you would like to think they would because, you know, every other significant technology market has kind of followed similar stuff. I mean, some have gone faster and some have been steeper curves and so forth. You know, the hardest part about this one, and, you know, granted, every major technology change we've seen, you know, there are limited supply of the people that know this stuff really well.

13:57And, you know, they cost a lot of money and therefore, you know, the intelligence is not spread. But it eventually happens, right? It eventually, you know, whether it's through training or whether it's through, you know, just diverse, you know, kind of diffusion of knowledge and so forth. But again, I think that the timing of it's really hard to predict because, you know, again, getting access and having access to the tools is very expensive. Like there isn't the, you know, immediate equivalent of open source. There's not the immediate equivalent of like cheap hardware, which we haven't had that for 20 years.

14:33I mean, like none of the trends over the last 20 years have, you know, have been really expensive. They've been like, I can get pretty inexpensive access to stuff and there's no limitations to me being able to get access to stuff, whether it's free tiers or open source software. I think that stuff will eventually kind of come down, but we don't know what the timing of that is. And if we don't know the timing of that, it's hard to sort of predict, like, it's probably not hard to predict what the future will be. But, you know, you feel a little bit like Simon Wardley where he always went, well, I know what the future was going to be, but I'm just never going to tell you what the timing is because I have no control over that.

15:08I feel a little bit of that same way of like, I'm pretty sure I know where this stuff's going, or at least I have a general sense of where it's going. but the timing of it feels so hard to predict just because the economics of this are so much different than anything we'd seen before. And maybe if we have a big crash and somebody's able to buy everything up pretty cheap because there's a bunch of GPUs that got stranded, then maybe that gets back to economics and timelines that we're like, oh, okay, cool. I get what the dynamics of this looks like. But that's the hardest part right now for me. And we've been doing cloud for 15 years and we draw parallels to cloud all the time.

15:44This one feels harder to, you know, kind of just map this with the way we used to map things.

15:49Aaron: Agreed. Agreed. And I actually want to close on one additional comment on the model customization and kind of futures there. Cause I had a theory there for a little while and I completely admit it was like a hot minute of like, okay, model customization, that's going to be like an entire category and it's going to be products and it'll move up the S curve. And I was actually following and some companies and man, the names are escaping me right now. But anyway, within a couple month period, all of them got bought and then open AI had their like a fine tuning API. They deprecated it. You know, Amazon came out with forge last year.

16:27Aaron: I haven't heard anything since the announcement and everything moved on to kind of FDs in the market. I don't know if everyone just kind of realized to your point, like maybe it's too early in the market. Like maybe it is just simply too early in the market and the skill sets just aren't out there. It's just not something normies can do right now. Yeah. I mean, it's, you know, sometimes, you know, the, the path to getting there, you know, you're like, Oh, this, this should be amazing. We should get there. And I mean, you know, the path to getting to anything really new is typically very, very ugly, right?

16:59It's a lot of blowing stuff up. It's a lot of getting in the dirt. It's a lot of, you know, two steps, you know, one step forward, two step backs. And, and yeah, it's, you know, if, if you, if you were to have told somebody a couple of years ago, Hey, the future of, of enterprise AI, given, you know, what you think you can do just for your first couple of chat GBTs is going to be entirely dependent on this army of FDEs, quote unquote, you know, that, that are really just consultants. You'd be like, God, we didn't learn any lessons from the last 20 or 30 years or whatever. And, but you know, that's, that's sort of the nature of, of what these things look like.

17:33So it's, I think the, I think the one really big unknown and, and Ben Thompson sort of talks about this a little bit, but again, he doesn't have any timing predictions. He's like, there is going to be somebody who's going to figure out what the AI native application or the AI native business model looks like kind of the analogy he always draws is like the internet went on for like five or six years until Facebook came up with the feed, you know, the algorithmic driven customized feed, and that changed the whole internet. You know, it's like, we really haven't figured out what the new experience of this whole thing is.

18:08And maybe when that happens, then the economics change and certain things simplify. But until that's kind of found, then yeah, it's a lot of the stuff's going to look like stuff we did before. Sure.

18:18Aaron: So I'm going to close this out with one additional. So we've been doing these on some of the shows. I'm going to ask Brian a question completely off the cuff. He's not prepared for. So that SAP event you went to, was that the one you broke up in Van Halen? Yeah, I did. I was, yeah, I was, I was the reason David Lee Roth, you know, second time he came back and the band kicked him out. So that's another story for another day, but that was a good time. Yeah. So Brian literally got to see his last one, which, oh, by the way, everyone's like, oh, they broke up. It's like, yeah, they broke up yesterday at the SAP conference.

18:48Aaron: That's a That's right. That's right. It wasn't even a real show. All right, man. Well, good show. I think we talked about some stuff. I think we're starting to find our stride with these shows. Folks, if you've got topics you'd like us to talk about or, you know, you want to come on and give us your two cents, let us know. Hit us up. Show at the enterpriseaishow.com. Love to hear from you. Love to get your ideas, your feedback. And Aaron, with that, we're going to wrap this one up and we will talk to everybody next week. Thanks for listening. Check us out at theenterpriseaishow.com for past shows, newsletters, and all things Enterprise AI.