

13 Oct 2026 · 56m · 19 chapters

The episode argues that enterprises should “rent” generic AI models but “build and own” specialized AI that differentiates their business. Manos Koukoumidis claims off-the-shelf frontier models are inefficient for specific workflows, don’t create owned intelligence, and can be taken away (e.g., model deprecations or providers competing with customers). He also stresses privacy/security and control over data retention and deployment.
Manos Koukoumidis, founder/CEO of UMI. Background: previously Google Cloud’s GenAI lead; bootstrapped the cloud PaLM effort that later moved into DeepMind as Gemini. Earlier built chatbot ideas at Microsoft (2015–2016).
Specialized models can be 5–100x cheaper while matching or beating frontier models; enterprises need AI sovereignty and continuous improvement; use AI to automate AI development (“cloud code for AI”).
bank customer-support ticket triage model (JSON outputs, evaluation vs Sonnet/GPT, improved F1/Fraud/category errors); mentions Kersor and Intercom using specialized approaches; analogy to databases (Postgres) and Linux for control/flexibility.
Written by AI. May contain mistakes.
Discussion on why companies should avoid using generic AI models.
“Manos Koukoumides is the founder of UMI, a platform built to help companies create, deploy, and continuously improve specialized AI models.”
Exploring the reasons enterprises should build their own AI models.
“There's multiple extremely important reasons why it makes sense for enterprises not to rent the most critical tool for their success, but to build and own their own tool that is specialized for the job.”
Examining the challenges and risks associated with using rented AI tools.
“and differentiation, you don't have that if you're just prompting the same model as your competitors.”
Drawing parallels between renting AI and historical tech trends.
“or because they are just deprecating it.”
Manos shares his background and experiences leading to UMI's creation.
“because again, all the same reasons, more flexibility to customize and optimize in your use case, lower costs, better control.”
Advice on when to use off-the-shelf models versus custom solutions.
“So you're at the moment before even 2017 when the transformer architecture and whatnot came out kind of playing with these ideas.”
Learn the decision-making framework for choosing AI models based on company needs.
“When should they be routing to several different models?”
Explore the importance of building proprietary AI for competitive advantage.
“Because quite often we say build on your AI.”
Discuss how specific firms are customizing AI for their legal practices.
“not the generic one that costs you 10, 100 times more than it should.”
Dive into the features of a new platform designed for specialized AI development.
“There's this continuous improvement element, this auto scaling deployments, which is very interesting when it comes with scaling with GPUs and whatnot.”
Learn how automation and AI can streamline the AI development process.
“I tell people right now, it may sound novel, but I think we're going to be looking back after a year and say, duh, why did it take somebody like Umi to propose that?”
Experience a live demonstration of building an AI model from scratch.
“Usually the way I describe the tube for people is if you are an AI researcher, likely you want to use the open source, even though there's actually some researchers that are using their enterprise platform.”
Learn about the iterative steps in training and evaluating machine learning models.
“This is for the step, for example, where it already built for me the test data and everything and it evaluated the baseline model where it said for your classification task, a QEN 3.5, 4 billion model is good enough.”
Understand the deployment process and performance insights of AI models.
“you have full flexibility and transparency for every step.”
Explore the future of AI integration in organizations and its impact on structure.
“It doesn't take you more than a minute if you know what you're doing.”
Explore how AI is transforming organizational structures and roles.
“But I'm curious, can you compare for me what it looks like in your organization today versus Google when you were there?”
Discuss the cultural shifts and dependencies brought by AI in teams.
“So they're kind of fitting in these purpose built roles, but they can do way more stuff up and down the stack.”
Learn the significance of balancing AI harnessing with model improvement.
“But then we put a lot of effort to make sure we use AI quite aggressively, but at the same time in a way that it will be more maintainable and we know what's happening.”
Understand why owning AI may become critical for success in the future.
“to action on the world in the best possible way.”

Get the gist, explore an idea or ask about something said.
On Talking AI, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Anytime in history, whether it's for technology or non-technology, when you care about doing the best job possible on something, you care about quality, when you care about cost, when you care about controlling your success, you don't go rent somebody else's generic tool.
0:17Matt Paige: Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Paige, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI. Every company says its data is a strategic asset, and many then send that data into the same handful of general purpose models their competitors can call to. And the answers may be useful, but the intelligence never becomes something the company owns. Manos Koukoumides is the founder of UMI, a platform built to help companies create, deploy, and continuously improve specialized AI models.
0:54Matt Paige: Before we meet, he was Google Cloud's Gen AI lead, bootstrapped the cloud palm effort that later moved into DeepMind as Gemini. And his thesis is simple. Rent the intelligence that is generic and own the intelligence that differentiates your business. And today I want to test where that idea holds, where it breaks, and what an enterprise actually has to build to make it real. Manos, welcome to Talking AI. Thank you very much, Matt. Very excited to be here and discuss about this with you. Yes, this is going to be a good one. I've been looking forward to this, but let's jump right in. When a company uses an off-the-shelf frontier model for a workflow that, let's say, differentiates their business, what's the bigger issue?
1:34Matt Paige: The exploding cost of how expensive the top-tier frontier models can be or the fact that the intelligence technically never becomes their own? How do you measure those different risks and variables differently? Yeah, I would say all of the above and more. And that's the interesting thing. that is not one reason. There's multiple extremely important reasons why it makes sense for enterprises not to rent the most critical tool for their success, but to build and own their own tool that is specialized for the job. And as you mentioned at the very beginning, yet most of them are still not doing this.
2:11But this is changing very fast. But let me start one by one. What are the main reasons? Why, as you also mentioned at the beginning, we don't tell enterprises you should just go build your own AI for everything. It's not a matter of building and owning the most AI, but a matter of the AI that is most critical for your success, that differentiates you, that makes you what you are. That's the AI you should own. Or the AI, make sure you have a specialist AI that affects your balance. So let me take it one by one. The first one is quality and differentiation of the service you provide. If enterprises want to innovate, which should be at the core of every enterprise, not just tech companies like us, you need to constantly be compounding and improving that technology that makes you better.
2:57And increasingly, it's becoming clear that the technology is AI. And if you do believe that it's going to be AI and you're just prompting the same model as everybody else, well, you're not going too far with innovation. You're pretty much using the same model. You don't have any mode. You have something that pretty much any other competitor can write the same or very similar prompt and build the same type of product. So when it comes to quality and differentiation, you don't have that if you're just prompting the same model as your competitors. And actually, as AI becomes the most critical component of what you offer, the less your differential becomes because the core or the biggest part of the product is the same.
3:34The second reason, as you mentioned, is economics. The thing that is so, what's happening right now, it's widely inefficient in the industry where enterprises are using like the Swiss knife for everything. They're using the model that was built to be good at everything for any task, right? Because it's a Swiss knife, right? It does everything, right? So I'm going to use that. And while this may seem like the easiest choice, as I mentioned before, it's widely inefficient. And it's a model that was trained to do a hundred or a thousand things when you only care about one thing at a time. And the thing is that you're paying for that.
4:14And what we have proven time and again, and many enterprises have proven independently, not even working with us, but on their own, like Kersor, Intercom with Finn and many others, is that you can have a specialized model that can be five, ten times, a hundred times cheaper and as good, if not better than these frontier models. Why? Just because it's specialized. and you don't need weights for a thousand things. You only do one thing. And the last reason which I think we didn't touch so far is about control of the technology. If, I think it's becoming clear that the technology is key for your success, if your future depends on it, you cannot be renting somebody else's tool that they can take away from geopolitical reasons what happened with Fable.
4:57They could take it away from you because they suddenly decided they're starting to compete with you. Like, let's say, Anthropic, doing drug discovery, financial services, coding agents. And suddenly they don't want to provide that model to you anymore because they want you to go directly to them to use it.
5:12Matt Paige: Most companies don't have an AI problem. They have a prioritization problem. There are hundreds of different places you could be using AI, but the hard part is figuring out which ones are actually worth investing in, which ones you're ready for, and what should actually come first. That's why we built Gen ROI at Hatchworks AI. Instead of basing your AI strategy on a few workshops or whatever happens to be in the room, Gen ROI uses AI-guided discovery to gather insights from across your entire organization. It helps us surface the themes, patterns, and connections that are easy to miss. Then our strategists work with your leadership team to turn those insights into a prioritized AI roadmap, weighted business value, feasibility, readiness, and what actually makes sense for your organization.
5:48Matt Paige: So if you're investing in AI, but still trying to figure out where it can actually create the most value, I highly recommend checking out Gen ROI. Go to hatchworks.com backslash Gen ROI, or just hit the link in the show notes. or because they are just deprecating it. That's a problem we have with a company we're working with, a bank, they say the GPT mini model is getting deprecated. We're being asked to switch to the new mini model. It's twice as expensive, not any better. And we have to do the work to change now. And last but not least, privacy, security. You know, if you don't own the model, you don't control where it's deployed.
6:19You don't control data retention. So again, it's all these reasons, you know, quality and differentiation, economics, control. And as I tell people, the answer that you should build and own your own AI when you care about this, it's not new. Anytime in history, whether it's for technology or non-technology, when you care about doing the best job possible on something, you care about quality, when you care about cost, when you care about controlling your success, you don't go rent somebody else's generic tool.
6:52Matt Paige: You build your own specialist tool. That was my follow-up question right there, because I'm thinking, what's the equivalent to this? Is there a time in history in the past? You know, the obvious one, people think of renting something, they think of cloud computing. But I'm curious, is there another analogy to this point we're in right now? Because it's kind of interesting. You're going from, you know, using the API and using models and whatnot, effectively renting it. And then you have people doing stuff on their local machines. Is there something in the past you would equate this moment to, or is it just unlike anything else we've seen before?
7:28So there is definitely lessons from the past, but also you could say this is very unique. And actually, if anything, those lessons apply even more to AI. When, even two and a half years ago, when that was highly controversial, highly, highly contrarian. Now, more and more are starting to see this was, again, what happened with databases. their Postgres SQL has more market share than Oracle. Why? Because more and more companies wanted that additional flexibility and control. Operating systems, right? All the internet, all the data centers, everything is running on Linux, not so much Windows because again, all the same reasons, more flexibility to customize and optimize in your use case, lower costs, better control.
8:13And the thing that's interesting is that these arguments, even though they were true for databases, for operating systems and many more, they're even more true for AI because there's one technology that is amenable to being trained, to be specialized, a specific use case. It's AI, even more than operating systems, even more than databases. It's kind of the de facto thing that you do with AI. Yeah, so that's why, again, what's been happening for a while is very paradoxical, but I think it's a matter of time and it's already coming. The transition to the right strategy to the right equilibrium in the market, it's coming.
8:53Matt Paige: Well, I think it's super interesting too, because the thought of a company training its own model doing this, it just seems so complex and only reserved for the frontier labs. I think that's what a lot of enterprises assume right now. I'm curious, before we jump into that part, because I think it's super interesting, and the fact that you can actually build this stuff in the platform, was there a moment in time where you realized everything, the monologue you just gave me, was there a moment in time where you were like, oh, this, this is going to be different? Like, were you still at Google when this happened?
9:28Matt Paige: Was there like a moment you remember where you're like, okay, I need to go build this thing? Yeah. Yeah. So for what it's worth, it was at least a year before that GPT came out that I was arguing that he was the next, because I was owning all the natural language services at Google Cloud, that the next thing we should do is text to text followed by multimodal text plus image to text. That was all almost a year before the GVD came out. It was not called Gemini at the time. One of the very first customers in the very early days of building Palm, which unfortunately I cannot name them, they asked me, a major organization that I'm sure you would recognize if I were to say the name, they asked me, can we get the weights of Palm?
10:10And I said, unfortunately, this is not allowed by the terms of use. We can share the weights of the model with you? Can we distill Palm to create our own version of Palm internally so we can freely do whatever we want? It's like, unfortunately, it's not allowed by terms of use either. And that was very early. That was just, again, only a couple of months after ZGBT came out. And that was the first time that I started thinking, okay, why are these people asking for all these things, right? It's a big model, complex to build, complex to manage, maintain, do the inference but you know yet they're still asking for more and then that's what i started realizing you know what actually they're not going to be a one-off because it cannot make sense i mean we built so many ai systems before yeah these are bigger models more capable but this doesn't mean that specialization you don't get benefits in terms of quality cost right they may look as if they can do everything they can be good at everything but this doesn't mean they are exceptional or they have the right economics or the right control that enterprises need.
11:14So it was actually very early on. And that was to give you an idea, at least, I don't know, January of or February of 2023. Just again, just a couple of months after the GPT came out that I was starting to realize this. And yeah, we're still very deep down into launching problem. Yeah.
11:33Matt Paige: Did you think this moment, like where, where we are today, did you think this was going to happen at this scale? I'm talking just broadly because you know, you're in it before even the chat GPT moment, you know, around what, November, 2022, like in your wildest dreams, do you think we would be here today in terms of the capabilities of the models, the proliferation of it? I'm just curious if that was even in, in your bingo card. No, I'll be, I'll be honest here. I mean, I didn't expect it would be as big. If you were to go back and look at some strategy docs I was writing at Google, I was suggesting, hey, this would be the next thing we invest across all natural language AI services at Google Cloud.
12:12I thought it would be the best opportunity we had in front of us, but by no means I was foreseeing the scale that such a technology would have. That being said, it was not also the first time I was building this technology. I had built something like ZGPT in 2016 at Microsoft, after an executive review with Chi Lu, who was reporting to Satya Nadella. I did an executive review presenting some ideas to him after winning the one-week hackathon that Microsoft had. I said, hey, what you're presenting is very similar to this effort we want to start. You should join the team and I'll give you headcount.
12:47And that's how I joined that effort, that we were building a multi-model, multi-model, open-ended chatbot. You could answer questions about everything. We even did embedded-based retrieval back then that I even proposed and prototyped myself, almost what people call RIG right now. But the thing was, and to give a lot of foresight to where it's due to all the executives back at Microsoft in 2015 and 16, we called it as an experiment. We used to refer to it as an experiment because we're not sure if the technology is there to make it successful. After almost two years, it was proven that, hey, the technology is just not there yet.
13:23But for me, that's why six years later, I was like, you know what? I could see how people were reacting to such technology six years ago when it was not as good. Now I could see from internal research at Google how much better it had become. And I think we should do this again. For me, that was the thing. We should do this again. We did six years ago, premature. Now is the right time to do this again. And again, that was before ZZVT came out. Unfortunately, yeah, we didn't. That is wild.
13:47Matt Paige: So you're at the moment before even 2017 when the transformer architecture and whatnot came out kind of playing with these ideas. Exactly. We were using at the time for generation. uh i don't know if it's like 70 80 90 it was double digit million parameter lstms they were not transformers at the time to use yeah interesting so and we're going to get into for those that are listening on uh apple or whatnot you may want to switch to youtube in a second because we're actually going to get a look at some of this live but before we do that give me an executive like an executive a decision tree like when should a company call just for an off-the-shelf model?
14:28Matt Paige: When should they be routing to several different models? When should they be fine-tuning an existing one? Or when should they be like training something specialized? Because I think this is where a lot of companies just get stuck in the minutiae of like, okay, where do I apply what? Yeah, no, I'm glad you asked. Because quite often we say build on your AI. But I don't want to be misunderstood that you should just go build everything yourself, right? I think there's two rules I often say. I think the first one, perhaps the most important one, which is consume, rent what is genetic, and build and own what is your differentiation.
15:05To be more specific, let's say, for example, you're a bank, since we're working with many banks. If you're doing coding, just use Cloud Code. It's a really exceptional model at coding, and you're not the only one writing coding. Any other organization across any other vertical, It doesn't have to be a bank, writes code. And Anthropik has done an excellent job, and us others as well, to create models that are highly capable specifically for coding. They put so much effort specifically for coding. It's almost a specialized model, almost for coding. So just use that. But if it's how do you do financial analysis, calculate risk or process mortgages, things that you are the expert on, you should be owning that AI.
15:51because how well you can execute that is what will sooner or later differentiate you more and more from your competitors and give you the competitive advantage. You do not want to be consuming Anthropic or Gemini or GPT for any of these tasks. Many issues, right? First one, it cannot be that you're abdicating your right to innovate and you're waiting for them to give you an improved model. when you are the domain expert on the field, you have all the data tropes and all the data operating that business for a long time. You just don't want to do that. Also, your competitors may not do it and then you're going to be in trouble.
16:31And again, you have the expertise, you have the data, you should be building and owning that AI, not letting Anthropic or any of your competitors catch up with you. You should be building that your specialized intelligence, your specialized, your most important IP and mode. The other rule is besides differentiation is, again, if something affects your budget a lot and your margins, you need to be using the right tool that is specialized for the job, right? I mean, not using a model that costs you 100 times more. And this means you use the specialized tool. If the specialized tool, the specialized model doesn't exist, you should build it.
17:08Quite often, this rule ends up collapsing with the first one because if you're doing something at high volume and affects your balance, it's likely that it's also where your differentiation often is. So you should already be building that model because your differentiation is there. Or likely somebody else has built a specialized model like coding and then you just go use that. So anyway, say rent, what is generic, build and know what differentiates you or what costs you the most. Make sure you're using the right tool, not the generic one that costs you 10, 100 times more than it should.
17:43Matt Paige: Yeah, that said perfectly. And just reading it back too, There's differentiation, there's like the volume of the task at hand. And then there's like this complexity element as well. It's kind of this maybe trifecta of some of those factors. And I'll say for listeners, just based on when these episodes launched, we had a mod from RiskVantage on prior, and he's literally, you know, PhD data scientist, head of AI, starting his own company in the legal field. And he's, his co-founder is like a domain expert in legal. and they're tackling this specific use case around procurement and legal and whatnot.
18:17Matt Paige: But they literally are doing what you're talking about. I need to connect them with you because they are fine-tuning and training their own specific model, like literally very tailored to their use case. And it's speaking to exactly what you're talking about. And I think this is like, this will be the future, very common in the future, I think, as we digs. Yes. The only thing I would say there is, let's say, for example, you are Kirkland and Ellis. should you use Harvey's, Legoras or the starter that you mentioned or should you build your own AI? Yeah. What's your take there? I think likely you should be building your own AI because especially for somebody like the size of Kecklin and Ellis, you should definitely be building your own AI if it's a smaller size.
19:03Matt Paige: Even Harvey and Legora, right? I mean, I don't know. They don't have their own models either. I don't know if that's in their roadmap, but you're bypassing another middleman, essentially. I don't know when the day will come that Anthropic may swoop them and say, no, actually, we are providing a better legal model and enterprise are going to them. Because if they are a thin wrapper on top of Anthropic, yeah, I don't want to minimize what they're doing. but the harness or some application and prompt around the model is not going to be a very robust mode. And we can get to that in a little bit later, but it won't be a very robust mode.
19:43That's why to give credit to Kirkland and Ellis, they came out their sermon a couple months back and said, there's many models out there, right? There's the Harvest, the Legoras, you know, all of this and many more, what Anthropic provides. But we are an exceptional legal firm, at least. That's why we think our customers hire us, not for the floor or the generic quality of legal services you can get almost anywhere. So we're not going to use the same generic AI as everybody else. We have the institutional knowledge. We have the capability. We're going to build and own our own AI. Because if it's going to be a key tool for us to provide better services, we need to have better tools than our competitors.
20:20It only makes sense.
20:21Matt Paige: Yeah, there's a massive precedence there with them doing this as a legal firm, which I think is just super interesting. Yeah. Yeah. So let's, let's actually get into the platform and I'll maybe set this up a little bit while you're pulling it up, but effectively you built this platform to where you can, uh, build a specialized model within it and monitoring in production. There's this continuous improvement element, this auto scaling deployments, which is very interesting when it comes with scaling with GPUs and whatnot. But take us through an example. And for those that are listening to this, we will narrate the experience as we go.
21:03Yeah, yeah. So just to give a little bit the motivation of what we built, UMI, the mission for what we built and why we left from building Gemini to build pretty much completely the opposite was that, again, enterprise should build on compound their own intelligence. It's going to be their most important IP and differentiation. pretty much what Satya said in his viral post in X just two months ago. And you should do that in a way that gives enterprises full flexibility, no lock-in, they have full AI sovereignty. That was our mission. Again, pretty much exactly what Satya said, what also Alex Karab has been saying a lot in interviews in CNBC and beyond.
21:45The thing, though, that was very clear is that, and actually we're still hearing from some of the largest organizations out there, is that it's hard. They are worried that building a specialized model for a task would take them months of effort. And sometimes, quite often, many of them would reluctantly suggest, hey, we're worried we may not have the right expertise to do that. Or maybe we have only a few people that can do that, but not enough for all the use cases. So, and we said, you know, building this specialized AI should not be that hard. There's so many things that you can do to help enterprise to build better AI.
22:21and why not use AI itself to help automate the development of AI? I tell people right now, it may sound novel, but I think we're going to be looking back after a year and say, duh, why did it take somebody like Umi to propose that? Why didn't we do that earlier? And I tell people, I don't know if it's because of lack of creativity of AI engineers, we automated everything except for our own job or false sense of self-preservation. I don't know what it is, but it only makes sense to use AI to automate AI development.
22:49Matt Paige: Do you see this as a form of recursive self-improvement? Because I think that the frontier labs are doing this, right? They're leveraging AI to kind of get in this loop. But do you see it as that, in a sense? Absolutely. There's a lot of elements of that, which is how can you make AI understand better where is it falling short, and how can it improve itself? Changing some hyperparameters in its training or changing quite often the data that it's being trained on, but offering that as a capability, as a frontier lab, you could say, that any enterprise can now own and operate. Sometimes we refer to this as the equivalent of cloud code for AI development, right?
23:28You can start with a prompt and say, build a model that does X and automates all the steps for you while still giving you full control flexibility to change every single part and component you want. Quite often we describe it as, hey, this is your frontier AI engineer. You know, all the experience we had from building Gemini we put into this AI. It knows what is the right process, how it can iterate to improve quality and automates all the steps for you while still giving you full control and flexibility. So that's the high level idea. And actually, if you want to happy anytime to also show you a quick demo about how it works and so you can get a good idea.
24:01Matt Paige: If you want, pull it up. Maybe we take a peek at it. And obviously there's probably some elements that we can't cover in the podcast given the time duration, but I think it'd be cool to kind of see what this actually looks like in reality. And while you're pulling that up, I'm curious, you're talking about open, and this is open source. I'm curious, like what went into your decision to open source this? It's already over what, 9 ,000 stars on GitHub. How does that decision play into this? What was your thought process in saying, okay, we're gonna open source this? And what does the, I guess the business model look like?
24:38Matt Paige: Yeah, yeah, great question. I'm curious. Yeah, to be very, very specific, what you see right now, you could say to be very specific, it's OpenCore. It is built on top of the open source foundation, an open source library that we have on GitHub, over 9 ,000 GitHub stars, as you mentioned, used by many organizations. And on top of that, we build this platform that you now see that exercises all the capabilities of the open source, but puts it into an enterprise platform that helps enterprises own all their assets, govern them, and the most important part, automate the development, which is the thing that makes the capability much better.
25:12Usually the way I describe the tube for people is if you are an AI researcher, likely you want to use the open source, even though there's actually some researchers that are using their enterprise platform. But if you are an AI engineer, that time to value matters, then definitely the enterprise platform makes sense, right? I would say the similar thing. If you are doing research on how can you write better code? Maybe you should write it yourself, not use cloud code, but if you're an AI engineer, you should use cloud code, right, to write your code. It's almost the same story. The easy button,
25:45Matt Paige: the harness for doing this. Exactly. Sometimes we say it's the first of its kind of AI development harness. It's because there's no industry precedent which was, yeah, we've been describing perhaps in three different ways at the same time, but yes. Yeah. Yeah. So the given, yeah, that's how it works. Yeah. When you go to me, has things like here's your data, your models, the evaluations, the deployments that you can do. But the key differentiation is, I mean besides having an end-to-end platform that you can really find is again, the equivalent of Cloud Code for a development. You can say build a model that classifies bank customer support tickets based on category and urgency.
26:26And you can start just with this prompt and hit go and then the agent that we have, we'll engage with you in a discussion to understand, make sure it understands fully the problem you want to solve and walk you through all the steps that you need to develop that model. So we'll see the first point was, okay, first, I want to clarify the requirements so I can build a model for you. What about these different types of categories? Because you didn't tell me what categories you care about. Or what about these different levels of urgency? And you can say, yeah, this looks good. You can give natural language to change things.
27:00And it will continue going through this process and makes sure it fully understands the problem and that starts walking you through all the steps of an AI engineer to build reasoning through this and figuring out
27:10Matt Paige: what things are needed to help kind of build a specialized model in a sense based on whatever prompt you give it in a sense right right just in the interest of time I'll show you another example which I had run before and actually I iterate quite a bit with the agent by the way all of this took me, I don't know, 15, 20 minutes of my time. But after he asked all the questions, he said, okay, here's the problem we're solving. The tasks automatically classify incoming bank customer support tickets. The input comes as raw customer ticket text. There's no pre-processing. And you're going to use this for an automated triage system.
27:44So you told me, or I suggested, that the output should be in JSON because it's going to be automatically processed. The categories are going to be this, and the different types of urgency levels are going to this. Here's the steps. First, you define the evaluators, meaning what are the different metrics that define success, you evaluate success, create your test data, evaluate the baseline model, which is what is the model you're going to start from, then create training data to improve that baseline model, fine-tune that baseline model, evaluate the model you just trained, and then see how you can keep iterating and going.
28:17If you want, we can go in all these steps, but it will say, it suggests you what are the right evaluation metrics, then it will tell you how to curate test data, it will do the valuations for you and just going super fast and I'll show you at the end some other key points because we could easily make that demo last minute, I don't know, half an hour if we want to. This is for the step, for example, where it already built for me the test data and everything
28:39Matt Paige: and it evaluated the baseline model where it said for your classification task, a QEN 3.5, 4 billion model is good enough. Here how it does in terms of, if you look at, let's say, for one score as opposed to accuracy compared to Sonnet. It's a 63 compared to 17. It's comparing to another model effectively. So you're kind of seeing how does it stack up versus maybe what I would use, Frontier model off the shelf. Exactly, exactly. And you can tell it compared with GPT, the big one, or Oppos. You know, you can ask it to compare it with you side by side with any open or closed model. But you can see the QN out of the box is not as good as Sonnet, right?
29:12It's quite lower. But it analyzes for you what are the problems. And then when it goes and creates training data, it automatically creates training data to fix these quality problems. it tells you how I will create this data actually I'll show you even later how exactly it shows you the detailed recipe sorry maybe it's here
29:30Matt Paige: so it's creating its own training data as well so you're not I guess I'm assuming you can supply your own training data that exists but it can also create as well exactly exactly so even before you start this process you can say you can upload any of your data structured or unstructured it could be just raw documents and say you know, here's example data that I have. Small amount of volume or large amount of volume. And if you have large amounts of volume, it will clean it up or improve for you. You have small amounts, it will use it as the foundation to build more and expand on it. So it can work flexibly with any of those data regimes.
30:06But anyway, it will create the right data. I will train it for you. I will then tell you what is the right way. Sorry, after you create the training data, let me go forward, forward. I will tell you what is the right way to train. If you should use parameter efficient fine tuning or some other approach. And you can see, you know, after you train the model, look at that. This is from 63, it went to 77. So it's better than Sonnet now on the category classification. For agency, it's still lagging behind. It's lagging behind Sonnet by four points. And so, okay, here's some issues that I still found.
30:37And I'll show you later how you can even go deeper into those quality issues. And I'll say, okay, what do you want to do? And so, okay, iterate to improve further. I selected that option. It's okay. I'm going to create new training data that addresses the failures of the model I just trained, fine-tune it again, evaluate, and then show you the results. And after I did all these steps again, this is the last run where you can see even the urgency F1 score. Now it improved from 64. That was the first train. 56 was originally. It went to 64. Now after training again, it went to 72. And now you have a model that is beating Sonnet and it's costing you literally, I don't know, 10 to 100 times less than Sonnet to operate.
31:20And literally all this process, it didn't take me more than 15 or 20 minutes. And this is the agent experience. Some key things I want to mention is that you have full flexibility and transparency for every step. When the agent tells you, here's how I'm going to evaluate, here's exactly how I'm going to synthesize data, here's exactly how I'm going to train, everything is recorded as a recipe. For example, for the data synthesis, this. That was the recipe that the agent used to synthesize the data. And you can go inspect this spec. You can use the agent here on the side if you want to tell it how to change the recipe for you.
31:58Let me maybe close again. Or you can even go hands-on keyboard. This is our open source definition that we have for the library in GitHub that you can have full access to all the parameters, everything so you can have full flexibility full raw control about how you change things so the key idea and that goes to the open source question you asked me before was that we want to build a product that will have the automation but it doesn't lock you in it works with any open model it gives you any parameter configuration any any option you want and even one day you can get the models the data and even the recipes for how every step was executed and run it, sorry, download and run it against the Apache 2 library so you can have full control.
32:41If you don't want to talk to me again, you don't have to. There's many other features I could show you. You know, for example, you can see this is the Sonnet model compared to the fine-tuned model, the version 2 that we built, right? I mean, you can show you all the detailed evaluations with the scores, all the confusion matrix, all the things typically you want to see. And a feature that strongly resonates with people is failure modes, where we tell you not just for the cases, the 50 samples that we tested, which one is failing, right, means if it's failing in multiple tests, but what are common issues that we found.
33:14For example, there is cases where fraud is getting wrongly categorized. And then it gives you an insight about what are exact examples and what are common problems that I expect to see. So this is how you build the model. Maybe to wrap up the demo, the thing I want to quickly show you is the following. Actually, maybe, let me quickly show you this. Let's say we also help you quickly test side-by-side the model that you just built compared to another model. For example, the fine-tuned coin model compared to OpenAI. So you can actually chat with it side-by-side just for those.
33:43Matt Paige: Exactly. It's a quick test. And let's say, for example, you say, I can't access my bank account. Just a quick sample. You can see this is the train model and this is OpenAI. You can see how much faster the train model was. and if you look at the cost I don't know, you tell me I think this is about two orders of magnitude cheaper than Lots of zeros after the decimal Yeah Okay, now very last one to complete the demo one thing that is very important also Satya said it in his post is when you build this AI you should have it compound as your IP and that's actually something that we launched just last week where when you have a model like this that you have built you can create a deployment minute, just take a couple of clicks.
34:26You say, okay, deploy model, you select it. Once you have deployed the model, like for example, this one, we show you graphs. Let's say, for example, you can see the last three hours of your traffic, how many requests were going through, what was the, you know, all the typical performance stats. But the most interesting ones is the insights about what is happening in your model. You can say, okay, out of all these requests I had in this window, I want to select this window and run failure mode analysis to see what are the common, how, what was happening in production? Where was my production model failing?
35:00Actually, there's one that I ran literally a couple hours before our, before meeting here today together. And that was this one, where for about 200 samples that I issued, it shows you which ones were failing. Again, it shows you the failure modes, like for example, for category evaluation, there were cases where actually the ground truth was mislabeled. So the model did the right thing, but it was mislabeled. Or cases where, for example, very important ones that one should know where fraud cases were misclassified.
35:30Matt Paige: And this can be fed back in to further improve the model, effectively. Right on. And all it takes is you can click on them. You can even click on those as well and say, generate now a data set out of these failure modes. And then you can use that data set. actually just takes a couple clicks you say okay generate that data set um well actually maybe i want to clean this because it may it will take a couple time to execute everything um but you can go generate the data set and then add it to your existing training set or however you want to handle it and train so literally it can take you i don't know three or four clicks no more than two minutes of your time or even a minute if you have done before and you know where to click and you can immediately train a better model actually i did this also less than 24 hours ago or maybe just a couple hours ago just to show you and the model that I trained was this one.
36:23So after I trained the model and I deployed in production, I evaluated with my production data to see where the quality was. And that was the failure mode that you were also seeing before. It was based on, this is a sample out of those. And then I clicked on the button that says create for me, analyze the failure modes, right? And after you have them, click on the button, select all of them, created a data set, append to the existing training set, now train again, literally four clicks. It doesn't take you more than a minute if you know what you're doing. And then it trained for me a better model that you can see again, the scores from 84 became 93, from 76 became 86.
37:01Again, this is not a cook demo. It was a real use case that we just once sorted into a demo to show you today. So it was not something that, you know, we had to try 10 times just in case we can get some good numbers. to show you today, it was exactly as it works.
37:15Matt Paige: So it's, and this is really cool. I mean, it seems obvious that, you know, just using AI in this way to build AI, it feels like the way I'm working now every day with everything else. So it just logically makes sense. But I'm curious, like when it comes to, you say you built the model, you're deploying it and you could actually, I'm assuming do that in the platform as well. What does that look like? Are you then accessing this model you build via API that you can use in other ways or run on your own hardware, I guess, whatever you want to do in a sense? Yeah, great question. So it can work in a couple of different ways.
37:47For one, after you have trained the model, again, it just takes, we say one click, it actually takes two or three, but you can click and say, here's the model, deploy it. You can type a name for your deployment if you want to call it something. You can select the number of GPUs you want to deploy it. You can say five GPUs, so you can say zero to 10, in which cases if there's no traffic You go into your GPUs, they cool down, you don't get charged. If there's a lot of traffic, it can scale automatically all the way to 10. This is one way where we do one-click deployments for you and you get an OpenAI-compatible endpoint.
38:18So you can literally, literally, no exaggeration, move from your GPT, Cloud, or Gemini endpoint to an endpoint powered by the specialized model that took you 15 or 20 minutes of your time to train. It may still take you hours, but only 15 or 20 minutes of your human effort into a new endpoint, and you have all that. Now, because again, we strongly believe in AI sovereignty, we make it possible so that you can extract and download all of the assets. You can download the model weights and run them in your own local hardware with open source library that we have on GitHub, we discussed before. Your data sets, the recipes for how every step was executed so that you can have full control.
38:59So because if you ask me what an enterprise would have, do you need to have all four? Because if you don't have all four, you're still getting locked into somebody's platform without you realizing or without them telling you.
39:11Matt Paige: So I'm curious, let's, you know, take a step back and look at this. So this open source future and potentials becoming more and more obvious. And I could see, you know, maybe this started, I used to exist with Hugging Face, but kind of a marketplace where people are using a tool like this, building their own specialized models and selling them or open sourcing them or whatever it may be. What do you think the future looks like between open source and the frontier labs? Because some people think that, oh, the frontier labs are screwed and open source is going to take over. Others I've heard, which I think is more logical, is along the lines of both will live, right?
39:49Matt Paige: And open source may get the vast breadth of, you know, token usage. But on the monetary side, that may be where the frontier labs still live because they're still on the frontier building the best of the best. but what do you think that future looks like? I'm curious between open source Yeah, so yeah, I don't think it's a you know we shouldn't say there's a false dichotomy as you mentioned I think there is a future for both close frontier models and open models If anything I tell people for consumer AI likely these frontier models make a lot of sense especially if you're building something like ZZPT you want a model or Gemini app or something that can respond to any question about anything yeah, you need something that understands pretty much everything but again for enterprises that have very specialized use cases internal institutional knowledge again it's their differentiation it's key to their success it only makes sense for them to build on compound their own intelligence as we've been saying for a while as also Satya and Alex Harf have been strongly advocating very publicly and very loudly lately so I'm very glad they are so and at the same time you know also for some enterprise use cases where companies like Anthropic build highly specialized models like cloud code, I think it makes sense to...
41:04So I think there's going to be, again, between consumer AI or different specialized models that these frontier labs develop, again, like coding, that they're going to still see a lot of usage and success because the market is expanding. They're going to still most likely see their revenue increasing, but not at the pace that many people are anticipating because it will make sense for more and more enterprise use cases to be built on top of open source AI. So they can be both the oracles and the Postgres sequels.
41:35Matt Paige: And then you look at NVIDIA too with Nemotron. They don't like to hype it up a lot because they're biggest customers on the frontier labs with just another player in the space. So I'm curious about this. Like we just saw, you know, what you're building and I'm assuming your team is small like most native organizations that are starting up today. And I always go back to the Jack Dorsey hierarchy to intelligence and what he's building and just rethinking what an organization looks like. But I'm curious, can you compare for me what it looks like in your organization today versus Google when you were there?
42:14Matt Paige: And how do you see the future of the organization evolving as AI just proliferates? Any cues into how your teams are using AI, how you're structured, anything that's interesting that folks listening can take from? Yeah, yeah. So I think in that article you mentioned, Jack Dorsey makes some good points. For example, I do agree that if you have AI, most of the things are now recorded, like meeting notes. Everything gets recorded with AI transcription. Then the information sharing role that the manager will have gets reduced. That being said, I think what information sharing managers do is a little bit more nuanced.
42:55They don't just pass any information up and down. So I think it's a little bit more nuanced what perhaps is in that article. But overall, what I see, which I think is, you could say, perhaps a little bit more obvious, is that whether you're a software developer, designer, product manager, or manager, CEO or non-CEO manager at an organization, you can be made more efficient. You have tools that can help you become more efficient. So before, if you're writing some amount of code, now you can route a lot more. If you're designing some features, now you can design many more and a lot faster. If you're managing people, you could actually manage more people.
43:31I do think there's a lot of human elements that cannot be scaled as easily.
43:38But I can definitely see that a lot of the work that I was doing as a manager, I could get a lot of help with AI to be able to support a larger team. That being said, The other thing that we see is a little bit of some sort of collapse of functions. For example, the most critical features are still designed by our designer because he can do the best job possible, better than AI. Is she using AI in her process?
44:07Matt Paige: Yes, he is. He is. Okay. I mean, he's using it, I think, with Figma. That's the element where he has the taste, the overused term taste, but I'm sure he has that in a sense of where he can apply that and get more from a context. Exactly, exactly. And it's not like he gave a blank check to Claude and said, now go design this new product. He built the design system, I think as he calls it, the whole framework. and then Claude has a structured environment that fits our brand, fits what he likes and what he thinks is the right design approach to go build more things that are aligned with this. As if you have a junior designer and you tell them, here's the guidelines, now go design this new feature based on these guidelines and these components that already been designed for you.
44:56And that's how he can now, he's much more productive. And even now for small features, what I said about the function collapse, for smaller features where it's a little more predictable how the design would be. Engineers don't wait for the designer to tell them how exactly to do it. They go and build it. And if then our designer, Konstantinos, has any vehement reaction to it, he may tell them, okay, now we should go change it and fix it. But there's so much more that we do, again, because Konstantinos can scale up his own work, or engineers for the work that CZ can just directly get it by consuming the same design framework that Konstantinos has put in, but at least they know how they can operate on their own.
45:38We have ML engineers that go write a lot more front-end code to complete the end-to-end experience of the features they're building. Before, they wouldn't go build front-end code because they would be so slow and they wouldn't like the idea, perhaps, of writing a lot of front-end code. But now they do.
45:53Matt Paige: Yeah, I think it was Boris Churny that was talking about this on a recent podcast or talk or something. But he was talking similar to what you're saying And it's like almost he's seeing this convergence around function or purpose where, you know, there was, I forget the terms he used, but like there was the maintainer, there was the builder, the prototyper. So they're kind of fitting in these purpose built roles, but they can do way more stuff up and down the stack. But they have like, they have, they still have a lane, but it's not specific to like, I'm a front end developer. I only write copy or I, you know, only do this, you know.
46:32Exactly, exactly. That's why the monitors have become a little bit more flexible. And again, especially when it's not the most critical design or the most critical product decision, the most critical front end to develop that needs to be designed in a highly efficient, specific, safe way. And it's more about, okay, let's do a little bit more of the same. yeah and again ml engineers go do that front end based on the design framework from constantinos fed to cloud and they're using the ui components built by the other front end engineers but cloud helps them reuse that to build exactly what they need yeah do you see any downsides to this or
47:07Matt Paige: or negatives yeah in terms of organizations you know i can see some over dependence maybe in the i don't know i'm curious if you see any negatives either emerging now or you think will emerge There is definitely. First of all, for one, there was, which I don't know if it was a negative, but it definitely got some cultural adjustment within the company to say, hey, you know, it's not just front-end engineers writing front-end code. It's not just Konstantinos dictating every single pixel in design, how it's going to be, right? Our designers, Konstantinos are designer. It's a wonderful guy. So it took some cultural adjustment there, but everybody, let's say, that's the right way to do things, right?
47:46We can be all more efficient. The other one, let's see, in terms of downsides, there's definitely some dependencies being built. We have, though, pushed hard engineers to say, don't just submit code for a pull request for a code review when you don't know what the code does, right? I mean, you can say, Claude wrote that, I don't know what that code does, right? This is not okay. So I think if you manage that, and also if you have these coding agents operate within a framework that you define, and you put a lot of effort into how you organize your code so that cloud code is going to go reuse your components.
48:23It's not going to go duplicate and triplicate. I don't know what's the word to do. Four times, five times the same code with a small variant because it doesn't understand that it's the usable components and it can just replicate or adjust. You end up with a lot of inflated code that's hard to maintain, that people don't deeply understand, and then it becomes a problem. But then we put a lot of effort to make sure we use AI quite aggressively, but at the same time in a way that it will be more maintainable and we know what's happening. And any automation is happening within our control and the frameworks that we define.
49:00Matt Paige: Yeah. And maybe the last question, this kind of segues in. So I'm building my first iOS native app using Codex. This is what I do in my nights and weekends now instead of playing golf or doing other things like I used to. but it's very interesting to see it working because now it can access the simulator. It's running through and testing and actually going through the flow of something. It has its own built-in browser and terminal where it's actually going and doing things in there. And computer use has gotten so much better, especially when you're pairing it with a really good model. But all that to say, I feel like at times the harness is almost just as important or potentially more than the model itself.
49:44Matt Paige: I'm curious how you think about the harness versus the model and those two things working together. Yeah. I mean, you need the body, not just the brain. And that's why I see the harness as how can the model execute and interface with the world, right? So you definitely need it. The thing though, because I've seen in the news recently about, we asked before about model routing or harnesses and all these things, all the values in the router or the values in the harness, yeah, there's definitely a lot of value. There's definitely things, if you have a better hardness, you can improve the overall quality of the model.
50:18That being said, I think why many people were focusing on this and forgetting about the brain is because they had the misunderstanding that adjusting the brain and improving it was impossible for them. So that's why I think you should definitely improve the hardness. Makes a lot of sense. Making sure you can fit all the right data to the brain and make possible for the brain to actually, to action on the world in the best possible way. But yeah, don't forget about training that brain because that's even more important than anything else.
50:51Matt Paige: Yeah. So to wrap things up, I want to try something new, calling it the three prompts. So it's kind of like a fire round. I'm going to ask three things. Let me know your thoughts here. So first one, and we're going to ask it to every guest. I'm just curious to see the difference in opinions and whatnot. But what's something, what is something cool or unusual that you're doing with AI right now, outside of, you know, uh, building, uh, models with AI with you. So anything interesting, this could be in your personal life, uh, within work, anything interesting you're doing with AI. Man, you already disallowed me from using the example I want to use, but I would say the most exciting, what's more exciting than enabling every organization to build its own AI using AI itself.
51:32Yeah, that was a given. So, yeah. So no, but other than that, I would say, I mean, like many people, right? I mean, I'm using now AI for many, many day-to-day tasks I want to do from helping me improve on an email I'm going to write to analyze a legal contract before I send to my attorneys because I want a quick response on something. So I wouldn't say anything terribly exciting, but definitely using it aggressively for almost anything. Just if you can make me a little bit more efficient, I'm definitely down for it.
52:07Matt Paige: So my wife's in the floral industry and they still write things by hand for stuff they have to do. And she was doing it the other night and it was driving me crazy. Looking at that, I was like, I can build a system to do all of this for you and she won't let me do it. Yeah. All right. What's one thing most people are getting wrong about AI right now? I think we discussed it. I think if I were to pick one, which is starting to change but most people still get it wrong is that that renting AI is the way for you not to fall behind that the way for you to succeed that the AI strategy for your organization should be renting and consuming the AI that somehow this few handful of people only can deliver to you if I were to choose that I think that would be the most important one because this then dictates all the strategy for the organization and yeah, I think whoever sticks to that strategy, it's going to be disastrous for the organization sooner or later.
53:11Matt Paige: Yeah, and this may be the synopsis of the conversation, but last one, what will feel completely normal in two years that just sounds strange today? Man, I think it's a good segue to say, yeah, building and owning your own intelligence. I think we're going to be looking back and saying, how come we thought that our most important IP it was okay for us to rent it or outsource its development to some other organization? Because right now this is happening a lot. FD companies are getting a lot of success. Our people are still renting these large models. I think we're going to make it back and say, how did we go so long and we thought that our most important IP, it was okay for us to rent it or outsource it to somebody else?
53:57Whereas the opposite of, no, we should build and own and continuously improve our own IP, our own intelligence, is going to look as the extremely obvious thing.
54:10Matt Paige: In two years, we will reset, reconvene, and see what the world looks like. Manos, thank you for jumping on and talking some AI. Where can people learn more about UMI and get in contact with you? Yeah, I think if you go to www.umi.ai, We have all information about the product, demos, a lot of resources, and also a link where you can go for free to get started, test UMI. Even credits are on us to get started for now. And yeah, you can see you're going to be able to build your own model literally in minutes of your human effort. That's awesome. I love it. Mattis, thanks for talking to me. I would be.
54:46Thank you very much, Matt. It was wonderful to be here.
54:48Matt Paige: Thanks for listening to the Talking AI Podcast. If you enjoyed the show, give us a follow or subscribe on your favorite podcast platform. And don't forget to leave us a review. We love that. For more info on Talking AI, visit TalkingAiPodcast.com.
55:27Matt Paige: partner, we embed Anthropic certified FDEs to identify high value business problems, build and deploy the solution, and put governance and security around it, then transfer the capability back to your team. If your cloud rollout is still mostly individual usage, check out how Hatchworks AI FDEs work at hatchworks.com slash cloud dash FDE. You can also find it in the show notes. Now back to the show.
Every company says its data is a strategic asset. Then many of them send that data to the same handful of frontier models their competitors can call. The answers can be good. The intelligence still never becomes something the company owns.
In this episode of Talking AI, Matt Paige sits down with Manos Koukoumidis, cofounder and CEO of Oumi. Before Oumi, Manos led GenAI at Google Cloud and bootstrapped the Cloud PaLM effort that later moved into DeepMind as Gemini. He left to build what he calls pretty much the opposite: a platform that helps any enterprise create, deploy, and continuously improve its own specialized models.
His rule is simple. Rent what is generic. Build and own what differentiates you or what costs you the most. Manos explains where that rule holds, why a bank should use Claude Code for coding but own the model that processes its mortgages, and why he thinks owning your intelligence will look obvious two years from now.
In this episode, you’ll hear about:
Key moments
Key Links
Mentioned in this episode:
GenROI by HatchWorks AI
Most companies don't have an AI problem. They have a prioritization problem. There are hundreds of places you could use AI, and the hard part is knowing which ones are worth the investment, which ones you're ready for, and what should come first. GenROI uses AI-guided discovery to gather insights from across your whole organization, not just the few people who make it into a workshop. Our strategists then work with your leadership team to turn those insights into a prioritized AI roadmap weighted by business value, feasibility, and readiness. Learn more at https://hatchworks.com/genroi