How Open-Source is Reshaping the AI Infrastructure Stack

0:08Aaron: Good morning, good evening, wherever you are, and welcome back to the Enterprise AI Show. This is your host, Aaron, and we are back to our interviews again after a short summer break. Today's topic is how open source is reshaping the AI infrastructure stack. We talk about Kubernetes, Kubeflow, and then dig into a variety of topics at this intersection of AI infrastructure and open source. Now, this one runs a little longer than usual, but it's worth it. And all of that comes up right after this quick break.

0:46Today's show is sponsored by Nasuni. There's a growing gap in AI right now between what's possible in theory and what successfully works at scale inside an enterprise. The difference comes down to unstructured file data. Many AI initiatives struggle because the file data they depend on is scattered, unstructured, and disconnected from where and how work actually happens. Nasuni changes that. It brings your unstructured file data into a single, secure foundation so AI, both generative and agentic, can access it with the context, governance, and performance it needs in production. Bring AI to where your unstructured data lives.

1:20See what it takes to activate your data for AI and request a demo at nasuni.com slash AI.

1:29Aaron: What does the outside world already know about your company? Cyber criminals could be seeing leaked credentials, compromised session cookies, exposed infrastructure, and even impersonations of your brand and executives. Nord Layer Intelligence by Nordstellar gives security teams visibility into those threats across the deep and dark web, data breaches, attack surfaces, and brand impersonation all in one platform. Find out what attackers know before they can use it. Visit nordlayer.com slash intelligence slash enterprise AI and use code enterprise10 for 10 % off your Nordstellar plan.

2:16Aaron: And we're back. And as we promised, we're getting back into our interview series. And to kick things off, we have somebody who's been in the industry a good bit, but newly introduced to myself. And certainly we've had some great conversations here recently. We are going to be talking about open source today. And we've talked about open source a good bit on the show before, but we want to talk specifically about how open source is reshaping the AI infrastructure stack. And for that, we have David Aronchek, CEO at Expanso. David, how are you doing, man? Good, how are you? I'm good, I'm good. So first of all, you have super interesting background.

2:58Aaron: You know, a lot of kind of accomplishments here. First managing PM for Kubernetes, co-founded Kubeflow, led open source ML at Azure. give everyone a brief introduction on how you became so involved in open source and the enterprise. Yeah, it's kind of fascinating. I mean, I really fell into it more than anything. I was originally a pretty hardcore Linux nerd way back in the day. I saw whatever, Slackware, Debian, when you used to get them on CDs. Not DVDs, CDs. Actually, I'm sure I actually probably have a boot on a 1.44 floppy somewhere. But then I left just the open source and my first two startups and went to work at Microsoft.

3:48And that's where I really got into enterprise software. And I was doing a bunch of Linux stuff for them even then as they were building out their own Linux strategy and so many folks were working on it. Then went off to do my third startup and came back out. and I joined Amazon, who was very, very big in Linux at the time, and then Chef, who obviously has an enormously successful open source platform. I was always a big fan of it because the idea of how you bring people together is so powerful around software. And software is one of the most collaborative things you can experience in the world.

4:25You can take a pull request from anyone in the world and it's just as good. But all that matters is the code, the quality, whether or not it achieves the goals and so on. So all of that was out there. And then I just absolutely fell, luckily, more than anything, into Kubernetes. Greg McCluckie, one of the co-founders, had identified me and we started talking. And he actually had to convince me to take on the Kubernetes job, mostly because at the time, people knew how deeply involved Google was in open source, right? They were core contributors to the Linux kernel. They had released a number of different amazing papers, the things that became Hadoop and so on.

5:06But they really hadn't developed an open-source project first thing from inside Google. You had Angular, and you had Android, of course. But those either come from external or they were very, very localized. So they didn't have the kind of reputation that they do now. And Kubernetes came along, and it felt like a science experiment, right? You know, Docker was at the time, and someone will probably correct me, maybe six months old. So I was like, oh, I don't know what containers, like LXC sounds interesting. And obviously, you know, zones. And there's been a lot of history to this. But anyhow, so Kubernetes is there.

5:44I see a lot of value in just the idea of this distributed orchestrator. Craig's like, come on. You know, I got six other projects I've got to go work on. you manage this for me for Google Cloud and launch the GKE project, which I did. And then, and then, you know, just obviously the train, the rocket ship left the station and, and it was just immediately obvious that I was never going to look back. After that, I was like, well, you know, Kubernetes is great, but this AI sounds like a thing. Somebody should work on that. Nobody wanted to. And I know you'd be like, really? Yeah. In 2017, nobody wanted to work on AI.

6:20We'd been through 14 AI winters. Everyone's like, AI is dead. I was like, I think there's going to be an AI thing for Kubernetes. I'm going to invent it with two other co-founders from Google and a bunch of people in the community. And so we created Kubeflow. And then, you know, obviously that, I did a bunch of work on that. Then I went over to Microsoft to lead OpenAI strategy. And then while I was there, OpenAI has the chat GPT3 moment and everything transformed from there. So, you know, I wish I could say I drew myself into it. I, I, this was a plan. I'm telling you, it was just a random luck.

6:55Aaron: Yeah. Well, sometimes that's the best, right? And I distinctly remember, yeah, the early days of Kubernetes when, yeah, to your point, it was like Docker, Kubernetes and Mesos were all like competitors for a hot moment until I mean, we, we, you know, in the team, we genuinely would have been totally okay if Docker or Mesos had picked up Kubernetes. and in fact we offered it to both of them right we said you take this you lead it go and neither wanted it now again that don't look at this as something kind of like oh they made this terrible mistake there were lots of strategies there and there was lots of good reasons to not pick it up oh we did you know it doesn't really like perfect fit for our platform not this that the other those are perfectly valid reasons who knows what along the way caused kubernetes to win versus these other once unclear but you know it wasn't a bad decision it was just like look we have our own thing and we like it and we control it it those were also open source swarm and mesos were both open source as well so it's not like if one was open source and one wasn't it just happened that one won no i agree so let's let's kind of because we've talked about kubernetes extensively over the years back when this was the cloudcast but i don't know that we ever did a show on kubeflow and maybe that was exactly what you're talking about like kubeflow was almost maybe a couple years ahead of its time if you will oh yeah tell me a little bit about that like okay the way i understand it is kubeflow was like kubernetes style orchestration for specifically for ml workflows right but that's exactly it that's exactly it and sorry no no please go ahead one reflection i'm just catching No, you are exactly it.

8:43So the year is 2017. I have now been leading Kubernetes and GKE, you know, having a great time, love the team. And I'm like, look, I think there's a space here. One of the things that Kubernetes really lacked was you would go in, and I loved giving this demo. I would go and I'd spin up whatever, a hundred no's on GKE in a minute for a customer. They're like, that is great. What do I do with it? I was like, whatever you want. Great. You don't have this. Go. Have your time. And that gap of like, okay, I have this general orchestra. What do I do on top of that is still, I mean, arguably still there, right?

9:20And so what I wanted to do was pick a workload that I thought was going to be interesting, valuable, demoable, all that kind of good stuff. And at the time, I don't think attention all you need have come out. I think that came out like at the end of 2017. but certainly ml was already enormous in particular areas a lot of people say oh you know i love to ask this question what's the first platform that has ever had a billion daily users using ml and people are like i don't know chat gpt and i'm like no the answer is google right google search in 2012 was all their indexing was all ml based right and they had lightweight inference models and things like that that allowed you to present those 10 blue links.

10:08Nobody knew about it. It had this wonderful interface, but they had this amazing infrastructure called MLX that coordinated all these things together. And they started releasing these as public papers. So go back and look at TFX from Google about using TensorFlow, a way to string together all the steps of the machine learning pipeline. You want to shape your data. You want to control it. You want to normalize it. You want to split it into training and test holdback. You want to tune your hyperparameters, et cetera, et cetera. There's always like many steps involved. Anyhow, and my thought was like, look, this is a perfect fit for Kubernetes.

10:48It's multi-workload. It requires loose coupling between the steps of the workloads. It is very hard to set up on your own. There is no simple way. It's like independently upgradable. Each component can be upgraded on its own, so on and so forth. This is a Kubernetes workload, and it's a complicated one that will enable us to take Kubernetes to the next layer up. So that was the idea. I had a lot of trouble finding the right people inside of Google to help work on it. And outside of Google, most internal people at Google thought that nobody was going to have their own ML pipelines at all. And that's where they went down some very, very innovative stuff that still is out there today around auto training and abstracting a bunch of things, try tuning and so on.

11:32and so forth. But I still felt there was a self-hosted thing. I talked to a bunch of customers and they're like, yeah, we actually have some self-hosted problems. We would love this. And so I ended up finding Jeremy Louie, who's now at OpenAI, who had created what's called a CRD. That's a custom resource definition for Kubernetes that allowed you to spin up a small TensorFlow component. And I was like, oh, what would this look like if we expanded this a little bit? Where it wasn't just that one component, I think it was just training. that one component, but we tied it to a notebook, a self-hosted notebook or a self-hosted inference.

12:07What would that look like? And he had some thoughts on it. We started like iterating on it. I then pulled in Vishnu Khanan from the Kubernetes team who had done a ton of work on GPUs and other accelerators. He's the third co-founder of Keyflow. And by September, we had like something running on our machines. We're like, look, this actually feels like it's there. i had gotten a talk accepted to kubecon that's back in the day when when i could get talks accepted to kubecon i believe i believe talks and kubecon and acceptance is a running

12:40Aaron: joke in the industry now yeah exactly uh and look i i was there i gave the first whatever four keynotes at kubecon and i cannot get my talks accepted so if you're out there like Like, it happens a lot. Anyhow, so I had a talk. I changed the title of the talk. I have no idea what the original talk was. I changed the title. I said, I'm going to make this about ML. I think it was called Hot Dog or Not on Kubernetes, which is a reference to the Silicon Valley show. And by that December, we had it launched. We had several cloud providers signed up for it. Red Hat was signed up for it. Bloomberg was signed up for it.

13:14They were all testing it out. They were all using it. And we got going. But it really was. Kubeflow was almost nothing at the start. It was a notebook, a training CRD, and if I remember correctly, an inference CRD. And those were very, very loosely coupled. It wasn't until the spring that the Cloud AI team at Google submitted a very large PR pull request on pipelining them together to make it very easy to take your data from, like once your notebook finished auto-triggering a training and triggering auto-training, triggering the inference. Now, obviously, there are many, many more elements. Kubeflow has just graduated from CMTF and so on.

13:55I still, by the way, think that this declarative rollout of pipelines is still an unsolved problem in a lot of ways. But it's obviously much further along. And Kubeflow has both achieved enormous success, of which I have very, very little to do. But also, it's inspired a lot of other people to build really inspiring stuff as well.

14:16Aaron: So let me ask you this, because I want to both relate it back to our topic and then ask a follow up question to it all at the same time. So I would say, at least as far as I'm aware, Kubeflow was was one of those early days, like you were saying, like, it's even pre GPT. And, you know, the big the big aha moment, right? And so like, how everything has advanced forward and specifically talking about AI infrastructure and open source within AI infrastructure. Tell me a little bit about like that generation, that first generation of infrastructure. And like, obviously we talked about what it got right, but like what it did, I won't say miss because it's hard to predict where this has gone, but like where, where is the current wave, like meaning like agents and inferencing at the edge and all of this, like where's it a little lacking?

15:07Aaron: Like where is the gaps and where's the growth potential? Yeah. Yeah, I mean, so my biggest thing, my bugaboo, and everyone's got their thing, is I want repeatability, right? I'm a big declarative infrastructure guy. I want to reduce the things that can be varied, imperative, and run time specific as much as possible. Because those are the things that are almost impossible to debug. They're impossible to roll out reliably and so on. And that becomes a really big problem. Now, what I was trying to do at the time was solve that problem via pure Kubernetes, right? Because that's what Kubernetes provides, right?

15:51It says, you give us a Docker with a specific hash and a specific deployment contract, and it will reliably, declaratively, roll that thing up for you. And that is good. But what I think we, one of the things we missed is that was not going to be enough, right? somehow you needed a language that spoke below that as well as above that so below i mean like what cuda drivers are you on what does your memory bandwidth actually look like all you know what is the os version all that kind of stuff how do you declaratively name that that was often a different tool and there were a lot of people who were like okay well we'll do this in terraform and terraform will take care of this on this cloud, but it will hand off to another layer.

16:39And I was like, man, we got to wrap this up together. Because even today, right? I just saw yesterday, Kubernetes or NVIDIA released a cluster readiness toolkit. And I got to dig in, but some very, very smart people have been working on that. Like some former Kubernetes, some former Mesos folks working on it. So you trust them that they're going to get it. But it's a perfect example where it's like, you know, if you go to cloud A, you go to cloud B, you go to on-prem C, and you're like, okay, they're all running whatever, Ubuntu 24.02, and da-da-da-da-da-da, right? We should be the same, and it turns out they're not.

17:14And just declaratively naming that makes things really hard. Why do people care? Well, one, a lot of your folks are just going to run into big challenges because they're going to be like, why did this perform this well here or that well there? Or maybe just longitudinally over time, right hey you know we ran this this training set last week and we're you know here was the results and we ran this training set today and here are the results and the results are different but nothing changed as near as we can tell and it turns out some random ass python package is different you're like okay right so that's a problem if you want my vision for the world this is how you get past things like scientific reproducibility this is how you get to actual intelligence being shared.

17:59Go and talk to any scientist in the world, and they do this weird archaeology when they pick up a paper. Even if it's got everything listed in there about all the details of the experiment, in incredible detail, something is wrong. They didn't measure the humidity, or, oh, they didn't realize that a truck drove by, or whatever. There's just so much non-reproducibility. And so I'm not saying that we're going to get there, But if you started to build things like this, if you said, hey, in order to run this scientific paper, this is specifically the entire environment that I used to set up, you can reproduce that.

18:38And if you get different answers, which you will because the world is not reproducible, at least you can narrow down where the reproducibility failures have occurred. And I am such a huge inspiration for that. But going all the way back, I thought to your question, what did we miss? we missed that reproducibility is still a massive unsolved problem that expands far beyond what kubernetes does but i would argue that we're just as bad today right like it may be even worse because so many solutions endpoints out there right now are just black boxes you don't own the data you don't know the training you don't know the version i i love you know i mean i use these ai tools all the time and they update every day every single day they update right i come in i click that button every single time oh update required just restart every single time you're telling me that i'm going to get the same results that i gave yesterday of course not right because there's no like transparency there's no way to inject determinism and things like that so we're still facing that today i thought we could get it done then we didn't couldn't and the failure there was twofold.

19:47One, it was hard to detail everything. But two, even if you couldn't detail everything, there wasn't a unified tool to take a bunch of metadata and say, okay, now I know how to blow this out. Today, we're running it on probably even worse because so much of the data is hosted, so much of the models are hosted, and we have even less reproducibility available.

20:06Aaron: Yeah. Well, okay. So this is super fascinating because coming from an infrastructure background And one of the biggest things I see is AI infrastructure, to your point, like a lot of the things you just mentioned, AI infrastructure today done properly is hard on prep. You know, and I'll go back to actually it has a Kubeflow tie in. My first, my days at Nutanix, you know, we did a Nutanix AI, you know, obviously on Nutanix converged infrastructure kind of AI thing. And it was based off of Kubeflow and open models and things like that. But the biggest problem we had was even with convergent infrastructure, damn, it was hard to get that stuff going.

20:49Aaron: And the amount of people that have the top to bottom expertise is hard. And so I guess my question to you is like, okay, where is the biggest opportunity for all of this? because in my mind, you know, open source and on-prem or not is kind of losing ground right now because of the closed vendor control platforms because of the simplicity like that click to reload, if I use that term, is both a blessing and a curse. Yeah. Look, I got to be honest with you. Tell me a little bit about that. Like, where's your head? You're putting your finger exactly on in, right? You know, I love Linux. I've used it for many, many, many years and FreeBSD and all these things.

21:31They have never, never gotten set up and reliable, like ease of use, right? And I don't know why. I mean, like arguably, like the affordance, the interface that they're looking for is different, right? I want to go in. I remember the first time I had something in the Linux core tree in 1998, something like that. I had a cheapo, you know,$40, which was incredibly cheap at the time, Ethernet card. And the version of Linux that I was using was wrong. The driver was wrong. It kept crashing when I compiled it. I went in, I found the code, I debugged it, and submitted it as a patch. It was just some stupid thing.

22:17And it submitted, and I'm still very proud of that today. I know that's probably long, long, long, long ago. It's like two lines ago. So that was incredibly useful for me because I'm a giant nerd. If I actually had a job, that would have been incredibly miserable, right? That's a terrible user experience. I just want to click a button and have it upgrade. So I would argue that a lot of these things, they're trying to serve two masters. They're trying to serve, again, and this goes to not just Linux, right? Windows and Mac and everyone has the same problem. Some people are just like, I have a job I want to get done.

22:49And some people are like, no, I want to solve this and make this optimized and so on and so forth. And by serving those two masters, you run into this challenge. Now, to your point about the clicked upload and things like that, look, 99.9 % of people have a job to do. And these things are simply functions of that job. Oh, I'm going to use ARI for this and I'm going to use Excel for that. I'm going to use whatever said or, I don't know, RStudio. I have no idea how to do this other thing. Those are just tools in the pursuit of their overall goal. The failure is that we ask them to go and understand what CUDA driver they're wearing, right?

23:28And so it's that mixing that I think ends up being this incredibly hard challenge. And if we wanted to make progress against this, it would be for those people giving them the ease of use to have that, hey, I have other jobs to do. I'm just going to work on this. But also give them the visibility or the transparency or even the voice to say, hey, I now want you to dip this environment that I'm using in Amber. and I want you to be able to reproduce this thing a hundred times over because I need this. I can't have a single thing change here because if it does, then that means I need to go back and redo everything.

24:09Having them be able to do those two simultaneously, that is the error, the problem for the error. And what I'll say is to just to up the level of challenge, and again, this is what we're working on expanse, but it's just generally a problem. Don't think of it as, people may have heard the term S-BOM, software bill of materials. There's something I'm really pushing on, another open source project that I'm helping to work on. called the data building materials or other mechanism for doing data lineage. Like, how do you not just detail this stack of the stuff you're running on, but detail all the input steps of the data that came in, right?

24:44How do you know what was in your training data? How do you know what was in your holdback data? How do you know, oh, hey, everyone, we're going to convert this from, I don't know, dollars to euros at this stage on this date. We're going to record that somewhere. Those are all inputs that happen even before you get your infrastructure stack like that. That's also part of the determinism that we need to do. So it really is like all of the above. You need to think about holistically. How did I create this artifact at the end that allows me to do my job? And what were all the things that came in? And some of them will be reproducible and detailable, and some of them won't.

25:26And that's okay. We just need to accept that some of them were left. Do not try and get to 100%. You will fail. If we got to 50%, I'd be ecstatic. Right now, we're at like 2%. So there you go.

25:39Aaron: I completely agree with your assessment, and it mirrors a lot of the conversations I've been having. But let me kind of move into, I'm going to call this kind of the grab bag section, if you will. I'm going to ask you a bunch of random questions here, because we have a somewhat common background, and I would love to hear your thoughts on some of these things that may not be as related to the AI infrastructure, but it's kind of related to everything we're talking about and doing in our day-to-days. So first of all, what's your thoughts on OSS models? And what I mean by that is today, a lot of us say, oh, this model's OSS, but it's really not.

26:10Aaron: It's open weights, right? And do you think there will ever be a truly OSS model run by Linux Foundation or CNCF? What would it take? And what is your thoughts on something like that ever happened? That is an excellent, excellent question. I know I talk long, so just don't know. Scratch your nose. I'll do it. When I go up to it. What I will say is this. You want to put your finger on something enormous, right? We have open weights. We have open source. I would argue even open source is not enough. Because think about what you're doing here, right? So open source means I have access to, go look at Stallman's like original thing, right?

26:50I have access to the code. I can use it according to this license, so on and so forth, right? There's a bunch of things involved there. But even if I have every line that Mistral or OpenAI or Anthropic used to train a model, every code, every hyperparameter, everything, that's not enough. That's not the model, right? The model is that I ran it and I ran it against this training data. So you can have an open source model that is totally useless because you don't have one byte of data or maybe you only have sample data or something like that so i'm not even able to create it i think a true open source model and i think this is something we need to define but like this is true would have the data and the full stack of infrastructure and the compute or the the code necessary to do that that allows someone to reproduce the open weights that were release this part of the model.

27:50That's what open source modeling would look like. I think that will be very, very, very challenging for people to actually do, right? Because ultimately, you are talking about petabytes of data that went into training. Now, again, maybe not directly, maybe it was a subset of sampling, things like that, but somehow petabytes of data produced those weights at the end. I think the best that we could get to reasonably would be maybe hosting some of that petabytes of data that allowed a third party to audit it and allowed someone to hash, create a hash or some other artifact from that data to say, hey, in this model, this is the hash of the data that went in.

28:35It's still not going to allow you to produce the open weights, but at least you have transparency into what the hash of that would in produce something on the other side? I don't know. But in the spirit of open source, I think we will have, you know, in the name, in the letter of the law, I think we will have open source. But I don't think that will be what we need if we really want to have the spirit of open source be realized.

28:59Aaron: Yeah, agreed. Agreed. Because I had a conversation with somebody, actually, you and I were at an event. That's where we got to know each other. And at that event, And actually, it was a side conversation, and it was interesting of like, hey, what would it take to do something like this? And it was like it would almost be like the Kubernetes moment all over again of like somebody would have to dump a whole bunch of stuff in. But to your point, even then, I don't think that's enough, right? No, totally agree. I mean, to your point. No, go ahead. No, no. I was just going to say, when you think about the values of open source, when you think about like what revolutionized, what that unlocked, it was the idea, right?

29:34Like, you know, Linux comes out and he releases the kernel and then there are whatever, 14 different distributions. And each one of those distributions where somebody's sitting around in their basement or their whatever office and be like, oh, I think we could do this. So I'm going to like take this. I'm going to fork it. I'm going to do something else with it. And again, that's the spirit of open source, right? If we were going to get there with models, that's what it would look like. It would be like, oh, here's all this raw data. Here's this code. Here's the infrastructure. I'm going to take that.

30:02I'm going to fork it. I'm going to make some tweaks in one or maybe all three of those things. And then I'm going to do this other thing with it. That's what open source would look like. That's where I think things get really, really, really hard. Because one or more of those things may not be available. Like you could say, hey, here's all the code for this model. But you need an NV, whatever, 72 running for three weeks in order to get it done. Well, it's effectively not open source, right? It is. I can go look at it. but I can't reproduce it. I can't compile it. I can't, you know, whatever it's challenging.

30:36And then I just don't know that we're there. Yeah.

30:38Aaron: Agreed. Agreed. So let me, I'm going to jump onto the next one here. AI native, and I'm using air quotes around it. You just, you can't really, you know, get that as much on the, the, at least the audio version of all of this, right? Like AI native, like it gets, it gets tossed around more and more, but like for an enterprise what does ai native actually mean from like the architecture point of view and like how is it different than just bolting ai onto the existing kubernetes cluster or you know whatever else going on inside the enterprise today like what's your thoughts on ai native oh gosh there's another long one i mean you know at the end of the day to be ai native i mean it It was like being cloud native.

31:22It was being whatever. It means that the majority, if not all of your workloads, your jobs, your everything that you do every day has the ability to use AI, right? Without having to re-architect everything. And in that, you say, okay, well, what does AI do? Like, truly, what does it do? What's the magic? The magic is it takes poor or unstructured input and it produces a meaningful output, right? and for a chat GPT, this was like taking my text and saying like, I want to go, you know, make my email sound like Freddie Mercury and then it could do that, right? So that's poor unstructured, produces something that is like reasonably structured at the other end.

32:02So that's what AI native is to me. Now, that said, what that means is, yes, having infrastructure that supports this, having your training, but even if you didn't do anything like that, it also means two other things. One, that the data that you're producing today is ready to be used by AI downstream. And that means having native APIs that know how to provide limited information so you're not revealing everyone's social security number but you're just querying what employees are working where, that kind of stuff. It means adding providence, lineage, tracking of the data as it moves through your system.

32:45So if I have 400 wind turbines in Iowa, every one of them, when they leave, you know, it's not just a bunch of comma separated values. It actually has all the information associated with each piece of metadata so that downstream AI can make better decisions because it's now got all that data painted. and it means as you pass it through the system, it's all recorded so that in six months time, when you realize you need this, if they're unavailable for you, not storing everything, storing everything or storing the stuff properly. So if you want to go back and recreate it, that's kind of, so that's a big thing, right?

33:21Really working on your data story, even before you get to AI, that's part of being AI native. The other side of it is, how do you build the systems to watch these non-deterministic systems, you know, results, AI, produce stuff. You know, if I go and query a web server six million times, I'm probably going to get the same result six million times, right? Every so often, maybe you've got a half a question or whatever, and it's going to get something wrong. Fine, that happens. But it's not going to generally, like, come back with, you know, the text of the Iliad. has the opportunity to do that. And what that means is that what we need to do, what you need to do to be AI native is say, hey, you know what?

34:08We're going to build in the reliability to our system to say, if you don't match the output we expect, we're going to have a way to fail elegantly. Or just say, all right, ignore, moving on, or re-request. Just that is also incredibly important as well. That's very different than the systems we built for many, many, many years where, sure, if the brakes on my car failed, it would know how to fail that. But like I said, if it produced a picture of, I have no idea, Tom Brady, it would not know how to respond to that. It may be a crashing system. Being an AI native means having a way to fall back elegantly when stuff goes sideways.

34:51So it really is, it's about where you can build these stacks, how you use them intelligently, but also what are the dependencies either before it lands on AI and after it lands on AI in order to handle this new and interesting system that we've built. That's what AI data means to me.

35:10Aaron: Now, okay, a follow-up. I do, I want to close this out on an edge question, but a follow-up on that before we get to that. Based off of what you just said and kind of putting a whole bunch of threads together here, if we want it obviously something that's you know llm based but llms are non-deterministic so we want something that is more deterministic and we want something we have better visibility and control over and is like you know i'm going to say is open source you know in all of this is there potentially a future in open source harnesses that then wrap around the model and help control the model.

35:51Aaron: Like I go, for instance, I mean, you look at all the different models that are making, excuse me, all the different harnesses that are making a lot of the models better. You look at Cloud Code and, you know, that notoriously, I think, kicked everything off. But like, I want to say it was Quinn. Quinn has a pretty interesting harness that they open sourced recently. And there's, you know, we're starting to see more and more of that going on. Like, what is your thoughts on how to solve the problem with open source? Is it a harness open source wrapper? Oh, absolutely. No, no. So these tools are, you know, all these models are just the start of what we're needing to do here.

36:31Right? The wrappers of them, the harnesses, are what begin to form the criteria, the deterministic criteria of inputs and outputs. And so, for sure, no matter how good your model is, it will never be anything more than an endpoint for input and output. And arguably, the majority of the work today, and this is pretty arguable, so I'm not going to take too spicy a take here, but I would argue that any model that we have today could be used to create 95 % of the value for the next five years. Like, really. It's about how you structure it, how you structure input, how you structure the output, how you iterate on it, how you update it, how you cache it, all those kind of things.

37:16And all that stuff is pure distributed systems, deterministic, you know, around the edges. Yeah, makes sense. Makes sense.

37:25Aaron: All right. So I want to close this out on a final topic. And this also kind of goes to a little bit more to your day job kind of thing here with Expenso. So I'm seeing a lot more both regulatory concerns, data residency concerns, really across a lot of industries, but certainly the highly regulated industries, healthcare, financial services, you know, things like that. how much of like edge and distributed compute is being driven by whether their ai performance needs governance compliance like tell me a little bit about that space right now and specifically the drivers behind them yeah i think it's a really interesting thing and you hear so much about this with sovereign AI and sometimes you're getting government regulations doing it.

38:15Sometimes you're just getting individuals caring about this. It really spans the map. Again, it generally is going to come back to who thinks they're going to get fired or sued or fail in some bad way. And Edge is a component of a solution for any of those, right? And the component of the solution is it allows you to execute portions of your job in a more specific way. And again, this is our day job. This is what we do with Expanso, distributed compute and distributed data pipelines and things like that, allowing you to run AI at the edge. We don't build any models. In fact, we have Mistral, we have Quinn, we have all these kinds of examples of running on-prem because all we want to do is help you run what you already know and love in those locations.

39:02But what it enables is those people who have these problems to say, oh, there is a solution here. It doesn't just mean handing all my data, handing all this proprietary stuff to an endpoint in the cloud and trusting that they have a zero data retention policy. I've got to get out of the acronym world. Or they trust that they're not using my models to train their own stuff on it. The fact that I can now get control of this is great. Now, I want to be really clear. I don't think that means stopping it, right? I don't think that means not using a hosted endpoint. No. What I think it means is being thoughtful about what occurs in each of the steps.

39:42What do I need to do on-prem to obfuscate, to clean, to whatever? What do I need to execute over AI locally because of bandwidth or regulatory concerns or whatever. And then what makes most sense for me to execute against that B300 in the cloud, right? Those things span the spectrum. Right now, there's only the one, right? There's only that, well, geez, I guess I'm going to use this thing in the cloud because that's the only one that's any good. Recently, arguably, you know, in the past six months, you've seen phenomenal results by folks with on-prem or other solutions, excuse me, self-hosted models, maybe not open source.

40:21I'm not going to fall into that target, but certainly self-hostable models that have started to change that. Again, it's not a question of whether or not they are better or worse across every benchmark. That is not required. But if they can do some components of the things you need locally and maybe entirely for some scenarios, then great. And people now have that as part of the option. So what I generally hear, you know, the answer to your question is I hear people being excited about this, adding it to their portfolio of solutions, but still in the same way they're not excited about being all in on the cloud, they're also not excited about being all in on on prep.

41:01They're happy to figure out the hybrid that fits best.

41:04Aaron: That makes sense. That makes sense. I think that's a great place to close us out as well. So David, if there's anyone out there that's kind of interested in that topic specifically, what's the best way they can get started? I'm infinitely online, unfortunately. I don't know how to break it. You can reach me on Blue Sky, Twitter. You can reach me on LinkedIn or so on. It's always on just about every place. It's just my last name is the easiest way to find me. There are a lot of us out there. I want to check out all these places, although in Blue Sky, I'm Iron Yuppie. Outside of that, you can email me.

41:37Again, it's just my last name. It's the same everywhere.

41:40Aaron: Love it. Thank you so much for your time today, David, and everyone out there. Thank you so much. Thank you very much for listening. We certainly appreciate your time this week as well. So if you could, please, wherever you get your podcasts, if you enjoy the show, please leave us a rating and a review as well. And we're always open to feedback through all of our socials as well. Both feedback on our episodes as well as ideas for guests as well. And so for Brian, who wasn't able to make it this week and myself, thank you everyone out there for listening. And we will talk to everyone next week.

42:09Thanks for listening. Check us out at theenterpriseaishow.com for past shows, newsletters, and all things Enterprise AI.