

5 Nov 2026 · 42m · 16 chapters

Total cost of ownership for AI agents, why “just call the API” doesn’t scale, and how to cut costs 10x while adding features. They argue that context, sandboxes, integrations, and misuse drive exponential spend; unit economics and eval-driven engineering are needed. They also discuss optionality (avoiding lock-in to OpenAI/GCP/one stack), in-house vs buy decisions, and hiring “AI engineers” by skills (software design, AI/agent tradeoffs, evals mindset), not titles.
Ioannis (podcast guest; discusses TCO, agent cost drivers, unit economics, evals, and hiring). Host (interviewer; asks about ROI, marketplace strategy, and engineering practices).
Endpoint calls from GCP/OpenAI don’t scale to 1k–10k companies; agent sandboxes store data and enable many integrations (Slack/email/Docs/Notion), making context expensive. Reduce frequency (like data freshness) and simplify instructions. Use a model router and granular eval suites to choose cheaper models per task. Build an app marketplace for education and chat-based surfacing to avoid rebuilding.
Written by AI. May contain mistakes.
Discussion on the peace of mind and scalability issues regarding AI costs.
“Way harder than I thought it was gonna be.”
Exploration of the hidden costs associated with AI consumption and usage.
“And you're talking doesn't scale to 1 ,000, 10 ,000 person companies.”
Analyzing when and how frequently AI should be utilized to save costs.
“Because it's free in terms of human consumption.”
Discussion on context management and the importance of simplicity in AI instructions.
“Do you really need to run every Saturday?”
Insight on determining valuable AI applications and managing costs effectively.
“In the end, what works best is the simplest.”
The importance of establishing a marketplace for AI skills and applications.
“and then he's the one responsible to distribute it accordingly.”
Discussion on the significance of evaluations in improving engineering work within AI.
“first objective of marketplace is educational so it shows what is possible okay yeah so that's my first objective the second on the product point of view You need to be able to surface this work in the chat.”
Exploring the challenges and opportunities in maximizing return on investment in AI.
“So if you build good evals, you can run and benchmark all the models.”
Learn how optimizing processes can lead to significant cost reductions in AI.
“For the same cost, if you have completely optimized your way of working, you can extract much more value.”
Explore how understanding model context can enhance efficiency and reduce costs.
“Are you that Hercules we're looking for?”
Discover the benefits of fine-tuning models for repetitive use cases and logistics.
Understand how to evaluate the cost-effectiveness of AI engineering projects.
Learn about the advantages of developing AI technologies in-house for flexibility.
Identify the risks of reliance on specific AI technologies and open-source solutions.
“can actually be then launched at scale to the whole organization, right?”
Discuss the evolution of AI tools and the importance of managing the tech stack.
“can we start building a first chat gpt assistant on slack in slack yeah yeah and it was just a way plug in chat GPT to Slack?”
“different roles and then you went from this to an agent where the agent could select the tools And now you went into a harness, right?”

Get the gist, explore an idea or ask about something said.
On Agentic Conversations (formally mlops.community), what are the key points of this episode?
Automatic transcript. May contain errors.
0:00It also causes there to be shared space between the client and the server. So certain features like visitations where the server wants to... I'm not going to be too long because I'm watching this keynote. But recently I caught up with Ioannis. We talked about the total cost of ownership in AI on a bike.
0:23Ioannis Zempekakis: Enjoy. Beautiful. Beautiful.
0:32Ioannis Zempekakis: One, two, three.
0:38Ioannis Zempekakis: Whoa, why are you going that way? Way harder than I thought it was gonna be. I'll see you on the other side. You're good at it, man. I don't do anything, man. We're gonna talk about your million dollar decisions. Are you ready for that? You're gonna be able to tell me about this million dollar decision that you did? What was your whole revelation? There are a couple of things, right? So you get lots of peace of mind if you go and hit an endpoint from GCP, from OpenAI and cloud. It's just that it doesn't scale. Because it's too expensive. It's too expensive. And you're talking doesn't scale to 1 ,000, 10 ,000 person companies.
1:21Yeah, it doesn't scale to, I think, in the end it doesn't matter to five ten thousand you can be one person company on the other side right who can do you who consume but who consume your product right so the moment that you go into a real consumption especially if you have hardness it's as greedy as it gets in terms of talking because actually you give your your harness a whole computer yeah with the sandbox
1:49Ioannis Zempekakis: Oh hey we got this sandbox that we're now paying extra for and it can do whatever it wants. Yeah so the sandbox itself is a cost, it's not very expensive, it's just a storage if you think about. But the agent now can store lots of things in the storage. And then you can connect, it can do 10 times more integrations than it used to be in the past. Because now it can run its own skills and it can run its own integrations. so it's five times smarter, the models are smarter, and then suddenly you need to pull out information from Slack, email, Google Doc, Granola, Notion, and check whether what we have discussed today has been properly translated into a PR or into a linear.
2:36And that can happen a hundred times. And then the cool thing is that it feels it's free. But actually it's not free. Because it's free in terms of human consumption. It's free in terms of human consumption and it's free in terms of... We've been consuming one way or another this type of technology, highly subsidized. Yeah.
2:59Ioannis Zempekakis: You're on the highly subsidized narrative. Anything in that? I think we are still on the high subsidized. If you put down the numbers from the unit economics, I think we are still... Not yet. And then what's happening is, okay, you build this product, you give it to the people, and then it's new. So everyone can try to do all the crazy things that they can think of, right? So they build apps, they create documents, they create presentations. They schedule things that probably they don't need to take place every five minutes. So there's also the misuse, right? And admittedly, let's say, first you release the technology and then you fix the experience, right?
3:47We are in this kind of idea at this moment because it's also very important to validate the technology. It's also changing very fast. So, yeah, sometimes you release it and then you don't make it clear. You know, you don't make clear problems. Do you really need this to run every week? Do you really need this to run every five minutes? Do you really need to run every Saturday? Why run every Saturday?
4:11Ioannis Zempekakis: But it was the same in the data world, right? You didn't need data to be fresh a lot of times. And a lot of ways that you could save money was by... Exactly. Reduce the frequency. I agree. It's just much more expensive. Yeah. And easier to set up. In the data world, there was only the data folks that were doing that. Exactly. And they knew. And that was the responsibility. And someone was looking at them, right? So now you have given this powerful tool to everyone, which by the way, a lot of them, they still need to put down lots of effort into... What can I do? How can I help me? And they can make the same up ten times.
4:51Because the first time they didn't know, so the second time they tried. And then this is a cool thing. They try to do very cool things that we didn't even think about.
5:01Ioannis Zempekakis: So you're pushing that down and that's cool but then you're also getting this repetitive nature because maybe the first time it doesn't come out as right and then throw away the whole code base. And then you do it again and then you do it again and then again. So there's this type of... And that gets expensive. And that can get expensive very quickly because the context is getting expensive and then you need of course to go into context management. and you need to go how do you do semantic search of your skills, so how do you feed it. In the end, what works best is the simplest. So give the agent as less instruction as it requires because then you are not overcrowded with lots of context and it works better even.
5:50So we went into that path and we went into that path first with cloud and OpenAI and then we find out very quickly that the costs are growing exponentially every day.
6:05Ioannis Zempekakis: And this is according to you with the subsidies. So you're expecting that eventually the subsidies will run out and it's going to get even more expensive. Exactly. You recognize that's not feasible? Yes, that is not feasible because lots of applications during this transformation are actually becoming viable. In the end, it's a unit economy, right? So if it costs me$10 to summarize my emails, I can live without, right? But if it cost me 0.0001, then that can become a big pipeline of 10 things that in the end it can make my life 10 times faster and better. So then you need to bring the unit economics in place.
6:51Ioannis Zempekakis: But how are you deciding what use cases are worth doing and which ones aren't? Is that where the hard work comes in? Is that why it's so hard? Yes, it's very hard. Because at the end of the day you can't have a judge looking at every use case and saying This one is worthy this one is not. No, there are two ways The one the traditional one you just keep the human in the loop Someone is the bug. So you make someone responsible for the budget
7:23Ioannis Zempekakis: CFO or somebody around Someone needs to pay exactly someone needs to pay the bill And that person you tell him you have hundreds of something of tokens, right? and then he's the one responsible to distribute it accordingly. That's one way and then there you can make everyone's life easier, you can explain to the users how much something costs. There's a little bit of education, there's lots of underground, you can become more informative. For example, hey you don't have to rebuild this, someone else has already built it. For that in mind we are building now and it's live a marketplace that you can select different apps that others have built, different agents that others have built, different skills that others have built.
8:10Ioannis Zempekakis: I've always had a lot of hate against that and I'll tell you why. Nobody actually wants to comb through the marketplace to see if they potentially can find their use case and then tweak a little bit of it. Maybe they send their agent out to do that and that that can happen but if it is so easy to just reproduce it on my end why wouldn't I just have my agent reproduce it in the exact way that I want it yeah I agree so I think the first objective of marketplace is educational so it shows what is possible okay yeah so that's my first objective the second on the product point of view You need to be able to surface this work in the chat.
8:59And not the user will need... So the user shouldn't go to the marketplace and start searching. If I ask something, build me a to-do list to track the teams, the velocity of my team on the PRs and the productivity, Dora metrics, right? So then it can immediately say, hey, someone else has built it. Boom, you get it ready. That could possibly work. That's awesome. work right and because then sometimes you know judging all these hundred things per dashboard it also takes effort right so in the past you had the person who's doing that and only that now i need to and then i have the space and dora and then i need to explain to the agent how i you know and all these things you know what man so if someone has done it i don't want to invest creativity on there yeah i prefer to invest creativity somewhere else right so so that serves that goal right on the market so we are also going that to avoid you know rebuilding lots of things but the biggest change was we built very good evals we have talked about this a year ago i think the first discussion that we had about having this discussion was on evals i remember we scheduled this call and you never show up
10:20Ioannis Zempekakis: And we were going to talk all about evals back in 2024 and then again in 2025. Now we're in 2026. And then it's evals, evals, evals. In the end, what you want is that you want to find a way to make your engineering work more professional. Yeah, and evals is a good evals is the way to make it more professional. So if you build good evals, you can run and benchmark all the models. There are different tasks, right? You have clustered your evals and then you can say for this task, I need this model, for this task, and this model. And this is where we moved now, right? So we're really getting granular though on A, the tasks and B, the evals with all the different models.
11:10sounds like a headache yes that's we go back it's hard yeah right so you get lots of things for free if you rely on just hit the API just hit the API the most expensive model and ignorance is bliss and by the way their SLAs are working quite good because any other Neo cloud today they are still trying to figure out and optimize and it's it's it's not GCP yeah it's all right so if there is one thing for sure it's not just
11:43Ioannis Zempekakis: had time to figure this out exactly they had time to figure this out we go on the tandem yeah let's go ride some bikes do we start yeah we're starting we're starting dude I you're much more maneuverable than Dex yeah man because with him you have to compete fully he was trying to lead bro he has decided the path is right yeah exactly i couldn't choose this guy's gonna go even with the steering wheel up here i could not go so but anyway you know what i was gonna talk to you about you know what i think what you're saying reminds me of is this idea of total cost of ownership have you heard that term thrown around Yep, yep.
12:27A couple of years ago.
12:29Ioannis Zempekakis: Yeah, exactly. And now it feels like what you're saying is we thought AI had this total cost of ownership, which was we would just ping an API and figure it out, which in a way it does kind of have that. But then you start to realize we're not prepared to pay that cost. We looked at the total cost and we said, I don't want that. Yes. And also I think there are lots of applications for intelligence that we are still scratching, right? Oops, that wasn't me. Nice, on the fly. What do you mean by that? Like we're still scratching the surface on certain things that... I mean, we are still in the ROI has not been maximized yet.
13:25Ioannis Zempekakis: And you said this to me before. ROI is a funny one because I've heard that the finance folks of public companies don't like to talk about ROI as much these days since they have to go and then say that on public earnings calls. So they avoid that term at all costs. and when you say ROI has not been maximized or found, you're just kind of saying like, we don't know if what we're doing with AI is actually providing value. Is that what I'm understanding? No, no, no. For the same cost, if you have completely optimized your way of working, you can extract much more value. yeah right so think it like that think it like even if it stops tomorrow the development of the models there's still been for next two years with the same level of intelligence we can do optimizations yeah that's what I mean right there's a lot of stuff that you can squeeze out yeah there's a lot of stuff that we can squeeze out of it there are lots of processes that you can remove and there are lots of processes that you can add right there are lots of things that you can do on on user experience to you know and and and and lots of engineering optimization right so we can we went go a little bit left here yeah all right there's some glasses oh no i understood left left uh all right don't worry just going with the flow you don't have an option bro i'm controlling yeah so that's the that's the value then comes when you really the easiest optimization is actually controlling the context yeah and then the models right so how you can find out mm-hmm which model do you need for which tasks yeah basic router basic router not the smartest router on the world just a basic a basic router and that'll get you pretty far and that's and that's where we are at the moment right so I mean with this approach we have reduced our cost 10x and we are just starting with this right so there's going to be much more optimization going forward but then at the same time we are also building 10 new features every month right so you are the pace is out running the optimization yeah that's i i think we are in this phase right so i think we are in the phase where you know you you you cut to one and then you see two popping out right yeah and cassandra isn't that your greek history Yes, isn't that what happened?
16:22Ioannis Zempekakis: Exactly. So we need Hercules. Are you that Hercules we're looking for? I'm not Hercules. Yet. Not yet. And then what is cool is you can start training your own model as we also start doing with LCM. Or then you start plugging into specific tasks. and that's all for this this whole idea of ROI yes don't stop puddling or you stop puddling on purpose on purpose because yeah I go down oh I think so I'll see you on the other side yeah we don't have to wait run jump on no no No, no, no, no, you don't want that. Okay. All right. Ready? The three. One, two, three. This is... Nice. Full trust.
17:28Ioannis Zempekakis: You just gotta go with me, huh? Yeah. So, and then you start fine-tuning smaller models, because lots of experience, it also requires a much smaller model in also in the sense of latency right yeah but you are then having to go and recognize which use cases are going to be done repetitively yes and how can we optimize that so that it becomes a workflow and we can use the smallest model to get us the end state that we're looking for exactly and a lot of these use cases are one-offs yes yes and no so if you if you take an organization that has logistics then it would probably pay off to start doing some post training or fine-tuning on some of these logistic traces right okay and then you expect to get similar performance maybe a little bit better at 20 30x the cost at 10x the latency or 5x the latency it's simply because the model is much smaller right so there is no smaller faster cheaper yeah exactly exactly and then you can't do this for everything but for example you can probably train a model and that's something that we are discussing for doing specific data analytics right huh because this is I mean yes it's never the question is probably never the same but everything around is exactly the same I think we're clear where are we going there we're going straight and then left you're saying the question isn't the same but the process is the same yeah the process is the same the data stays the same the the data data data databases right so you can you can have probably a track of 100 000 questions that have been already asked but does it need it that's the question i have like does the new we're going right here right pedal yep yeah big turn all right all right we did it and then we're gonna do another another one eventually because are you not just doing hard engineering problems for the sake of hard engineering at that point you know what I mean like does the is the juice worth the squeeze or is it just a whole lot of squeezing do you understand what I'm saying yes I understand training a model or rling a model going back to that total cost of ownership there's a whole lot of things that you need to do yeah to make that successful yeah I think the answer lies on them it's always related to scale right yeah so if you do it for five people then no it doesn't if you do it for five thousand data analysts or five thousand engineers yeah that they hit this every day then it does now have you can you potentially run this exercise to the cost of I don't know the end the engineer spend also three hours doing this instead of no but you can you can make some good you know you can run some good numbers and then you can see okay if we go that way with that type of extrapolation then it will pay off right so that's the first thing and then you're also building the muscle right so you are training your team and also there are lots of byproducts of this for example the building a model router right yeah so mill building a model router is one thing and you can buy it let's say yeah there's a ton of them out there yes but you need also for a model router you need also to have an amazing eval suite right and then you you go into the byproducts sometimes yeah the steps removed of what the main thing you think you need and then you realize wow if we're going to have success for this thing then you have to look a little bit exactly exactly but then yes it is engineering and and i don't think so that gen ai has been removed the hardcore engineering i think i think contrary because now you can build so fast you need the basis to be very solid so you need good infra you need good observability where do we go right or left left left left we're gonna cross here all right see if we do our right crossing how ready for this oh and then we go right or left left left left left left left into the park I'm trying to go into the park get a little but there's stairs there so we go right here we go right no there are stairs there I know that's not okay okay look at the lion though lying in a bear park this doesn't look like a lion no that's totally a lion no that's definitely not a lion if you just had a haircut can we stop pedaling okay so going back to it the muscles you build are valuable and it's much more beyond like the basic hey 100 % And then what happens there is a byproduct is that these muscles and these capabilities can actually be then launched at scale to the whole organization, right?
23:38So the moment that you find out how to get good evals, then you do them for everything. Then you do it for everything because every agent will require some evals one way or another, right so then you can package it and then and then and then enable everyone to do it right yeah and and then that that depends on the it's always on the scale in a scale of 30 000 employees like process ecosystem has at the moment this is incredible it can
24:12you can completely change how we do engineering and developing
24:15Ioannis Zempekakis: thinking there huh on that turn you got a little scared i heard it in your voice oh we're going left and so okay there's the cost benefit analysis that you're doing too to decide does this use case warrant us to go and try and do our thing on our own or try and create our own version of this internally, right? Yes, but also there's another thing when the technology change so fast you want to build some technology in-house. Oh, he said no. Keep paddling! Okay, I can do that. No bikes. Well, we'll do a lap. Ah, but we didn't know that. Yeah. So how did that person know, right? He knows the rules.
25:11He has done it.
25:12Ioannis Zempekakis: For some pull-ups? Yeah, let's do some keeping, right? Yeah. You've been here with Dek. I know, dude. This would have been a lot easier. Fuck. So. I'm not for the benching.
25:26Ioannis Zempekakis: So anyway, you were at... Wait, wait. Yeah, I remember. So I was telling that because technology is changing so fast, you also don't want to lock yourself in in something that you don't control right so there's also a value of optionality right and those are the muscles that you're building yeah so that's one of the advantages if you build imagine now you commit into a there is no clink clink clink there's no hole I don't want to do that to the kids on their shoulders to scare them So the... Imagine now if you commit to OpenAI or Clawd and then you're locked in because you built everything on it.
26:08Ioannis Zempekakis: Yeah, exactly. It's not so easily swappable. And I mean OpenAI is probably, let's say, the safest bet. But there are also lots of open source libraries out there. Imagine if you have committed to OpenClaw in December. probably around March, you have to scratch down everything and then follow Hermes. And now OpenCode or Pi code. And then these projects can also divert. So then you always need to think, okay, is it then do I convert back? So there's the open source, there are other small companies that they can pivot anytime also, right? So the fact that you a customer doesn't mean that they don't decide tomorrow to do something else and then what do you do with with the whole stack right well yeah even amazon or gcp can do that too yeah exactly yeah yeah yeah i mean there's so many products that they've released that eventually just died yeah i agree so so in that sense in that sense having uh let's say being uh not being the first that you adopt something it might also be if you want to scale yeah sometimes good option right yeah or you build it yourself because then you can quickly pivot and so if you think about a can we start building a first chat gpt assistant on slack in slack yeah yeah and it was just a way plug in chat GPT to Slack?
27:45Yes, as a way to talk to GPT-3.
27:47Ioannis Zempekakis: In 2022 or 2021? One month before chat GPT. And then we pivot from there into an let's say a web application where you had to select different roles and then you went from this to an agent where the agent could select the tools And now you went into a harness, right? And this is only realistically possible if you control at least a big part of the stack. Yeah. Because otherwise, what? You're just at the liberty of OpenAI doing that for you? Or if you use another startup or any other, let's say, open source library, there is always the risk that there is the other thing around, which is better, faster, cooler.
28:51And then you either need to commit to what you have and make a pull request and then pray that all the forces will align to get accepted. or you need to make changes to your stack but then you don't you cannot ensure easily backward compatibility.
29:12Ioannis Zempekakis: Well I guess what I'm thinking as you're talking about this is there's a certain amount of use cases that you're trying to figure out where to invest your time in to optimize and and you want them to be at a scale but you also want them to almost be like generalizable in a way so you were talking about the data analyst and it's not that you're learning a one data analyst workflow you're not optimizing that one workflow you're trying to optimize as much as you can in the data analysts like complete yeah the suite it's not just that one thing that the data analyst does right and so there it's it's more generalizable yeah it is use case by use case yeah how do you go through and you look at these and you recognize all right this one feels like somewhere where we should be investing our time and effort yeah good question there are some operations that they are critical for business right and those are probably those that they've been measured over the time they've been their data metadata do we step out here yes all right that's giving out don't fall all right one two three send it So the metadata?
30:37Yeah, so there are already some operations that... Actually, there are operations that you're investing already effort before AI. So customer support, data analytics, account management, optimizing your marketing budget, optimizing your logistics. So you know, right? So the business didn't change. right? So then you know what are the use cases that you need to invest and make them more scalable and then at the same time what you need to do which is also our strategy is that you want to give, you want to unlock all the other new things that are now possible right? And the way to do it is by giving a tool like Tocant where you can build apps and then everybody can build apps now yeah and then you can see people that before they need to have an engineering team now they can they can literally just go and build something themselves and then you can see probably over time how many other people are using it
31:42yes and then you are start getting new insights okay now this is how we'll see right so that's that's a more bottom-up approach right of course there's in the the top-down which is okay where the company wants to go right so where the company wants to invest where do you want to optimize and innovate well that wasn't me yeah it's crazy I just do a little bit left and then it feels like you go we're going right here yeah did I answer your question yeah so the other
32:21Ioannis Zempekakis: piece to this that I think is fascinating that we've talked about before and I don't know if you really have a clear answer on yet is who are you hiring and what kind of skills are folks expected to have because AI engineer means so many things now yeah right like what is an AI engineer even mean so you don't hire for for sure you don't hire for titles you hire for specialization in a specific area of your product. So what do I mean by that? It's probably extremely hard to find a person who has SRE experience so he knows his infra and platform and AI and evals, right? So the whole stack, right?
33:14Ioannis Zempekakis: yeah so what you're serving optimizing serving and exactly exactly but there are some things that you cannot discount and this is for me good software engineering design thinking around this right so how you can design good lean scalable solutions because the coding now more or less has been abstracted but how do you design your solution has not been right and how are you testing for that with a traditional old design system solution so you do a whiteboard design system and maybe you do more than one you might need to do two yeah just to really get a feel yeah well we're going to go left we'll go up here and then go which one there yeah so the crosswalk all right see if we can you You tell me.
34:09Ioannis Zempekakis: See if we can survive this intersection. Oh boy. Now we're on. Wait. Relaxation mode. Alright. Alright. One, two, three. Let's fucking go. Sorry. Did I go too hard on the pedal? No. Okay, so you've got the traditional whiteboard interviews. Yes, that needs to happen, especially now because coding itself doesn't say anything. You need to understand if you understand how to build things that they scale, things that they can, you know, they are not over engineering. Now over engineering is quite simple because you just ask, but then maintaining is hard. So it's smelling the over engineered code that Fable tries to give you.
34:56Exactly. And then if you ask what else I can do to improve, it just keep on coming and keep on coming. Right. So it never says, I think I've done enough. Let me go back. Yeah. Congratulations.
35:08Ioannis Zempekakis: Yeah, exactly. And then that's the first one. Then second, he needs to know AI. And for me, it's a very broad thing. What do you mean by know AI? Yeah, exactly. So he doesn't necessarily need to know how to train a cat boost or not even a neural network, I would say. not for what we do yeah there's another team in process that they are responsible more for training the models so that's not the thing but they need to have a good intuition and a good knowledge about how to set up an agent what are the existing limitations and when you say agent what do you mean you can pick up hermes for example right okay so can can i pick up something around why for example hermes has been a preferred agent over opencloth right okay have you tried and have you tested and what have you found out right so why opencloth stay right so what is the reason for it how do you build evals right so there's there's a little bit understanding all of that Yes, knowing AI.
36:18Ioannis Zempekakis: Exactly. That's my kind of health. Mushies. And then he doesn't necessarily need to, you know, go build evals, but he needs to know going forward, the more successful products in terms of engineering and scalability is going to be those that they have a very strong foundation, but then there's a lot of work that's being done from the agents yeah so he needs he doesn't need to engineer things if the agent can take care of it so you need to understand whether this person can realize the trade-offs do I build this how I used to build it five years ago or do I completely delegate this to be a decision of the agent right so do I build for example a static dashboard with front end to show some numbers or do I allow the agent to you know develop a custom canvas style right dashboard thing that can change over time right because the whole software is going much more into personalized software yep so if you build a very hard tightened product it's gonna be very hard then to decouple it yeah so recognizing that product sense yeah the almost expecting someone to understand that just as much as they understand the software side of things yes and i think the hardest part in hiring yeah is to find people that they know both because people that they are very very good at software engineering uh they have they are already having some years of experience and if you think about it AI was there before so then the question is okay where I find the person who is good at software engineering but then at the same time he switch means that he has experience in software engineering but then at the same time he's in love with these new technology that you want to try everything right yeah this is but I'm I'm sure that there's easier by the day.
38:24Yeah. Because everyone has tested it out.
38:28Ioannis Zempekakis: One, two, three? Yeah, one, two, three. All right. Come on. I'm going to give you some force. You gave me the little push to accelerate. Acceleration. There's your car. Nice. There's your car. Oh, that's my car. We've made it in one piece. I didn't expect that. You didn't expect that? I thought I would need to say sometime, hey, Nina, my wife my wife i love you please don't you're probably not gonna see me again where are you going i think we can park it here all right all right all right so okay we stop stopping stopping i don't see you stopping all right you want to finish that thought uh yes and then we can finish the finish it it's getting easier and easier but i don't know if it's getting easier to find someone now I think it's harder to find someone with AI engineering and then train him how to be software engineer because that requires a lot of grinding and with it with coding agents now it's much harder than finding software engineers that they are very open-minded and then yeah and then you plug them into a very heavy AI team and then they can style they can start you know smoking AI 24 7.
39:49Smoking out.
39:51Ioannis Zempekakis: What I've found is it feels like it is a lot of noise in what the title is or what like someone who says yeah I understand AI does that mean they understand how to code with AI they understand how to build agents they're using Hermes and they're they understand like the context window they understand the serving side they understand the infra side what does that even mean and so the lack of understanding and what or the shared concepts i think is the lack of shared content steps makes it much more difficult there are not job titles job description your job description really needs to be these are the five things that we mean yeah when we say ai engineer define our terms yes exactly let's go back okay ai engineer doesn't mean that you build an agent using Landcraft.
40:41Yeah. Right? And you know how to use cloud code.
40:44Ioannis Zempekakis: Yeah. Right? That's what I mean. Okay? And then, and sometimes this is an interview, right? Yeah. So we spend, we invest lots of time and talking to lots of candidates to find out who can be happy, right? Yeah. Because it's a different role than probably what they're used to, right? Or if they've done it, then they know. And so that's also the case, right? So you can you can also hire people that they are building similar products one way or another Then you need to check other things Look at evals, dude. The evals are what will get you. Like are they ready to grind out all these evals and that hard part?
41:21Ioannis Zempekakis: Man, it's been awesome. I'll let you go. I'm glad we finally did this. It was a long time coming Thank you. Let's do it again. Talk about evals. We can always talk about evals As long as you're ready to get back on this tricycle is tandem We're doing it.
41:41Nice. Nice.
What does AI really cost once thousands of people inside a company start using it? Ioannis Zempekakis, Director of AI at Prosus, has been running agents in production since 2024. We put that question to him on a tandem bike ride from a favorite Amsterdam cafe to Vondelpark.
Ioannis explains the decision to stop relying purely on cloud APIs, and why that approach breaks down at the scale of a large company. He covers why agent harnesses and sandboxes are as greedy as software gets, and why agents scheduled to run every five minutes repeat a lesson the data world already learned about freshness. He also explains how the habit of throwing away a codebase and prompting again quietly adds up. His bigger point is that even if model progress froze tomorrow, there would still be about two years of value left to squeeze out. The cheapest wins come from controlling context and running a basic model router, while ten new features a month keep sprouting like Hydra heads.
From there we get into when fine-tuning small models pays off for repetitive work like data analytics, and the eval suites that come as a byproduct of building in-house. Ioannis explains how owning the stack let Prosus grow a GPT-3 Slack bot into a full agent harness. He also explains why committing to a single vendor or open source project is a bigger bet than it looks, and how to balance top-down priorities with bottom-up building across the company. We close on hiring. Ioannis looks for system design over syntax, a nose for the over-engineered code AI models love to produce, and enough agent intuition to know when to build something yourself and when to let the agent decide.
Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm
Ioannis Zempekakis: https://www.linkedin.com/in/zempekakis
Timestamps:
[0:00] AI total cost of ownership on a bike
[0:48] The million dollar decision
[4:26] When everyone can build with AI
[6:51] Which use cases are worth the tokens
[10:46] Benchmarking every model on every task
[12:18] Total cost of ownership and AI ROI
[15:12] Context control and model routing
[16:39] Training and fine-tuning your own models
[19:41] Is the juice worth the squeeze
[25:26] Optionality and vendor lock-in
[27:30] From Slack bot to agent harness
[30:12] Top-down vs bottom-up AI adoption
[32:19] What AI engineer even means
[35:13] What knowing AI really means
[37:47] The hardest part of hiring