From AI-Curious to AI-First: Engineering Production AI Systems

0:28Welcome to the Deep Dive. business value most companies are seeing. The potential is, I mean, it's undeniable, but the reality on the ground for a lot of organizations is quite different. So our deep dive today is all about the engineering realities, you know, what it actually takes to build production-grade AI systems. We're using the insights of Armand Ruiz, right? He's the VP of AI platform at IBM, leads over a thousand engineers doing this stuff in the real world. Exactly. And his core assertion is pretty blunt. Becoming truly AI first is fundamentally an engineering discipline. It's not just research anymore.

1:01And the data absolutely backs him up on this. There's a 2025 McKinsey Global Survey. It shows over 75 % of organizations are using AI. Sounds great. Yet a staggering 80 % of those same companies report, well, no material contribution to their bottom line, their EBIT for their generative AI work. That's a huge production gap. It really is. So what's going on? What's causing this massive disconnect? Well, a big part of it is how easy it seems to create a compelling demo. You know, hook up a language model to a vector database maybe, and suddenly you've got something that looks impressive. Right.

1:36The dangerous illusion of progress, as Ruiz calls it. Precisely. It makes everything look much simpler than it actually is under the hood. He calls this getting stuck in slideware mode. Slideware mode. I like that. And if you zoom out, there's this 2025 survey from Rider.com. It revealed something pretty alarming. 42 % of C-suite execs admit AI adoption is actually tearing their company apart. Wow. Tearing them apart. Yeah. Mostly due to internal friction and siloed development. 72 % of executives reported that as a problem. Get all these siloed demo grade projects popping up and they mask the huge engineering effort needed for real reliable production systems.

2:15It's this weird paradox, isn't it? We can prototype incredibly fast, but that speed is creating a deployment bottleneck down the line. That's a great way to put it. So our roadmap for today's deep dive is basically to dissect Armand Ruiz's five-point strategic framework. He really believes this can bridge what he calls the great divide between AI hype and shipping code. Sounds like a plan. All right, let's cut to the chase then with Ruiz's first pillar, and it's maybe the most challenging one. Start with software, not just research. Now, for a lot of people, Gen AI feels like this magical new thing.

2:49So why is Ruiz so insistent that it's really about disciplined engineering? Why is that view so vital? Well, it's because these AI systems, especially the ones using large language models, they behave very differently from traditional software. Traditional code is deterministic, same input, same output, every time. Right. Predictable. Exactly. But AI systems are probabilistic. They're non-deterministic. Their behavior is learned. It isn't explicitly programmed step by step, and it can even sort of drift over time as the underlying data or model changes. Which means your old ways of testing just don't cut it anymore.

3:22Pretty much. Traditional QA methods are insufficient. You can't just write a simple unit test expecting one exact output string. Quality in AI requires continuous evaluation, constant monitoring, and importantly, iteration. Okay. And that's precisely why having a mature CICD pipeline, continuous integration, continuous delivery becomes absolutely indispensable. It automates the testing, the packaging, the deployment of the entire AI system. Not just the code, you mean? Not just the code. The prompts, the configurations, the evaluation data sets, all of it. It's the whole package. So it's less about picking a specific programming language and more about committing to this whole architectural philosophy, this engineering discipline.

4:04That's it, exactly. What specific technologies does Ruiz point to for building this foundation? And more importantly, why are they crucial for AI specifically? Yeah, his recommendations are very intentional, very much tied to the nature of AI systems. First, he mentions FastAPI for building APIs. Okay, APIs. Why FastAPI? Because an AI model itself is kind of amorphous, uncertain. But to be useful inside a company, it needs to be exposed as a reliable service. Fast PI helps create these stable, documented contracts. It effectively decouples the model's inherent uncertainty from the other applications that need to use it.

4:42Got it. Creates a clear interface. What's next? Second, asynchronous Python or async Python. Think about it. Large language model inference, getting an answer back, it can take several seconds sometimes. Yeah, they're not instant. Right. And if your application just stops and waits for each call, it grinds to a halt. It can't handle other users. Async processing lets your system handle thousands of concurrent requests efficiently. It switches to other tasks while it's waiting. It's really an architectural necessity for scalable AI. Okay, that makes sense for responsiveness. What else? Third, Docker for containerization.

5:15You know the classic, it works on my machine problem. Oh yeah, the bane of developers everywhere. Well, it gets amplified in AI because you have all these complex dependencies, specific library reversions, system packages, maybe even GPU drivers. Docker packages everything up the code, the libraries, the OS bits, ensuring you have identical execution environments everywhere, from the developer's laptop right into production. It eliminates a whole class of deployment failures. Consistency. Crucial. And the last one. And finally, back to CICD. The experimental nature of AI means you need to iterate rapidly.

5:51Automated CICD pipelines give you that velocity, but also a safety net. They test the whole system automatically, catching issues before they hit users, and dramatically speed up how often you can safely ship updates. It's really fascinating how these foundational software engineering principles become even more critical in this probabilistic AI world. It's not like AI replaces them. It demands them even more. That's a perfect summary. And this blend of software engineering rigor with machine learning understanding it seems to be creating this demand for a new kind of role, doesn't it? Yeah. The AI engineer.

6:23Absolutely. The AI engineer is exactly that, this emerging hybrid discipline. They bridge the gap between, say, data scientists who are focused more on experimentation and model building and traditional software engineers who build the application logic. So they're the ones actually making it work in production. They're the ones translating those promising models out of the research notebooks and into reliable, monitored, scalable API services that the rest of the business can actually use. Ruiz calls them the most critical talent profile for any company serious about being AI first. You know, there's this constant debate around Andrew Neng's point that the real value isn't necessarily in building the next biggest foundation model, but in the application layer built on top.

7:06How does Ruiz's engineering first philosophy connect with that? Why is building around the model the durable advantage? It connects directly. As these powerful foundation models become more available, more commoditized, maybe through APIs, the real lasting strategic moat for your organization isn't going to be owning the absolute smartest model off the shelf. Right, because someone else can just use the same model. Exactly. The advantage comes from having the superior engineering capability to build an efficient, reliable, scalable system around that model. That system includes your proprietary data pipelines, your fine-tuned agentic workflows, your optimized infrastructure that's much harder for competitors to copy than just switching to a different LLM provider.

7:50So the ability to ship shippable AI, as Reese puts it, that's the real differentiator. Ultimately, yes. It's a function of your engineering culture and your engineering capability. That's the durable moat. Okay, that commitment to shipping shippable AI really leads us perfectly into Ruiz's second big insight. This one tackles a concept that I think is widely misunderstood, the true nature of AI agents. He argues pretty strongly this isn't just a fancy chatbot. It's a fundamental architectural leap. What exactly defines that leap? Yeah, this is key. A production-grade AI agent isn't just about conversation.

8:25It's really a cognitive architecture designed to mimic human problem solving. It's not just a single monolithic call to an LLM. So how does it work differently? It operates on an iterative loop. Think, thought, action, observation. The agent thinks about the goal. It decides on an action using its available tools. It executes that action, and then it observes the result. And that observation feeds back into its next thought process. Okay, so it learns or adapts based on what happens. Exactly. It's this iterative, stateful process that fundamentally distinguishes a true agent from just a reactive, stateless chatbot that mostly just answers one question at a time.

9:03So it's not just talking. It's doing things and adjusting course. What are the core building blocks, the components, that make up this cognitive architecture he talks about? To build one of these, you need several distinct subsystems working together. First, you need a planning engine. It's like the agent's strategic brain. Strategic brain. Okay. What does it do? It takes a high-level goal like summarize these reports and email the key findings and breaks it down into concrete, executable steps. But crucially, a robust planning engine supports dynamic replanning. Meaning if something goes wrong.

9:35Precisely. If an API call fails, or maybe some data it needs is missing, the agent doesn't just stop. It analyzes the error and generates an alternative plan. It needs to have fallback logic built in. This is absolutely vital for production systems because you can't have silent failures. Right. You need resilience. OK, so planning engine first. What's second? Second are memory hierarchies. And this goes way beyond just remembering the last few lines of chat. It's more like how our own brains manage information. OK, intriguing. How so? You can think of it in layers. There's short term memory, like working memory, holding the immediate context, what the user just said, what the agent just did.

10:14Then there's midterm memory, which is more like an agent's personal journal or episodic memory. It stores a persistent record of past interactions, conversations, task outcomes. So we can learn from past mistakes or successes? Exactly. Avoid repeating errors. Remember what worked. And then finally, you have long-term memory. This holds the agent's enduring knowledge base, maybe distilled insights, learned procedures, and user preferences for true personalization over time. There's even academic research on things called memory operating systems to manage all this. Wow. Okay. Planning, memory. What else?

10:48Third is intelligent tool orchestration. Tools are how the agent interacts with the outside world. APIs, databases, search engines, code interpreters. They're the agent's hands. Makes sense. But it's not just about picking the right tool. It involves meticulous parameter preparation, formatting the data correctly for the tool's input. And maybe most critically, it needs robust error handling. What happens if that API is down or returns an unexpected error? Back to the resilience point. Absolutely. A production agent has to have predefined retry logic, maybe fallback strategies if a tool fails, and mechanisms for graceful degradation doing the best it can even if some tools are unavailable.

11:27That is definitely a far cry from just plugging an LLM into a chat interface. Yeah. And Ruiz even talks about taking this further, orchestrating armies of agents, mentioning frameworks like Microsoft Autogen or Crew AI. Can you give an example of how that might work? Yeah, absolutely. Think about a complex task like software development. Instead of one giant agent trying to do everything, you might have a team. A planner agent outlines the features based on requirements. A coder agent takes those specs and writes the actual code. Then maybe a critic or tester agent reviews that code, runs tests, and suggests improvements.

12:03They collaborate. It's like a virtual dev team. Sort of, yeah. And Ruiz's own example of agentic rag is another good illustration. He describes a meta agent that receives a complex query needing information for multiple documents. Okay. Instead of trying to read everything itself, it orchestrates specialized document agents, each an expert on one specific document or data source. They each provide their piece of the answer, and the meta-agent then synthesizes those partial answers into a coherent final response. That sounds much more scalable and robust for complex multi-source information. It really is.

12:36And the key takeaway here is that the intelligence or the autonomy of the agent isn't just inherent in the LLM itself. It emerges from the careful composition and orchestration of all these components working together, the planner, the memory, the tools, the control logic. So the LLM is like the reasoning core, but the architecture around it determines how effective it can be. Exactly. You can have the most powerful LM in the world, but if it's embedded in a poorly designed, brittle architecture, you'll get a poor agent. The system design is paramount. Okay, moving on to Pillar 3. This one tackles another area ripe with misconception, retrieval augmented generation, or AA.

13:13Rees argues the common view that ARRI is basically just vector search is an oversimplification that leads to, under quote, silent and catastrophic failures at the retrieval stage. Why does that naive ARRI, the kind you see in quick demos, fail so badly in real enterprise settings? It fails for a few key reasons, especially with complex enterprise documents. The typical approach is chunk the document into pieces, create vector embeddings for each piece, store them, and then do a vector search based on the queries embedding. Right, the standard demo setup. But arbitrary, fixed-size chunking often breaks things.

13:50It can sever critical context. Imagine splitting a sentence or paragraph right in the middle. The retrieved chunk might contain the right keywords, but lack the surrounding information needed to make sense. Ah, okay. Context gets lost. Exactly. Plus, relying only on semantic similarity, what sector search is good at, often misses things that require exact keyword matches. Think specific product codes, technical terms, or names that might not be semantically close to the query's phrasing, but are lexically identical. So it fails on both context and precision sometimes. It can, yes. And the dangerous part is the LLM is often so good at generating fluent text that it can take these irrelevant or incomplete chunks and weave them into an answer that sounds plausible but is actually factually wrong or misleading.

14:35The failure isn't the generation, It's the retrieval feeding it bad information, garbage in, garbage out. Okay, so if that's how naive ARAG fails, what does a sophisticated enterprise-grade ARAG system look like? How do you engineer it to overcome these limitations and actually get reliable results? Getting ARAG in the enterprise requires moving up the ladder of sophistication with more advanced retrieval techniques. First, it starts with thoughtful chunking. Not just arbitrary splitting. Right. Moving beyond fixed sizes to contextual chunking splitting documents along logical boundaries like paragraphs, sections, or even table rows.

15:11An even more advanced method is something called sentence window retrieval. Sentence window? How does that work? The core unit you retrieve is a single sentence, which gives you high precision. But then you also provide the LLM with a window of sentences around that core sentence to give it the necessary context. Best of both worlds, potentially. Interesting. Okay, better chunking. What's next? Second, and this is a really significant leap, is embracing hybrid search. This acknowledges that pure semantic search or dense retrieval using vectors isn't enough. Neither is pure keyword search or sparse retrieval using older methods like BM25.

15:48So you need both. Exactly. Hybrid search combines both approaches. It runs dense and sparse retrieval in parallel and then intelligently merges the results. This combination handles a much wider range of enterprise queries, robustly queries that need conceptual understanding, and queries that need precise keyword matching. Okay, hybrid search makes a lot of sense. What else improves it? Third, and this is critical for quality, is adding a re-ranking step. This usually involves a two-stage pipeline. Two stages. Yeah. The initial retrieval, hybrid search maybe, is optimized for speed and recall finding all potentially relevant documents, even if some aren't perfect.

16:24You might get, say, the top 50 candidates. Then you use a second, more powerful, but computationally more expensive model, often a cross encoder, to re-rank just those top candidates. The cross encoder looks at the query and each candidate document together in much more detail to assess true relevance. And pushes the best ones to the very top. Precisely. It dramatically improves precision, ensuring the handful of documents actually passed to the LLM for generation are the most relevant ones. This significantly reduces the chance of the LLM getting confused by irrelevant information. Got it. Chunking, hybrid search, re-ranking, anything else?

17:02One more absolutely crucial piece, systematic evaluation. You cannot treat our gag as a black box. You need quantitative metrics specifically for the retrieval component. Things like hit rate, precision at K, mean reciprocal rank separate from evaluating the final generated answer. So you can diagnose where the problem is, retrieval or generation. Exactly. This allows you to systematically diagnose and fix issues in the retrieval part, which is often the root cause of poor IG performance. Wow. That is a lot more involved, a lot more engineering than just embedding some chunks. It really sounds like there's this whole rag sophistication ladder that organizations need to climb.

17:38Absolutely. The source material actually describes it that way, a ladder from level zero, which is that naive reg up through levels incorporating hybrid search, re-ranking, contextual chunking, all the way to level three, which is agentic reg, where the agent itself might refine queries or orchestrate retrieval. So it really drives home the point. Effective IoT is an incremental engineering process. Requires genuine information retrieval expertise, not just plugging in a vector database. Couldn't have said it better myself. All right. Pillar four. Sure. This focuses on LLM system design, and Ruiz makes a strong claim here.

18:12We're past the point where just prompt engineering and maybe some fine-tuning are enough. He says it's now fundamentally about composition and orchestration. What is this new engineering discipline he points to, LLMOPFs? What's it all about? Yeah, LMOPs. It's essentially a specialized domain that extends traditional MLOs and DevOps principles, but specifically tailored to the unique challenges that come with large language models. Okay, so how is it different? What makes LLMs need their own ops? There are a few key differentiators. One is prompt management. Prompts in these systems are almost like source code.

18:46They dictate the model's behavior. So LLM ops involves version control for prompts, testing prompts, managing libraries of prompts. Treating prompts like code. Makes sense. Then there's non-deterministic output evaluation. As we discussed, LLM outputs aren't always predictable and quality can be subjective. So LLM OAPS needs sophisticated evaluation pipelines. These might combine automated checks using other models, rule-based heuristics, and crucially, human in-the-loop feedback systems. Right, evaluating fuzzy outputs. Tricky. Very. And the third big piece is composition and orchestration. LLM OAPS isn't just about deploying a single model.

19:23It's about managing these entire chains or agents or complex workflows involving multiple LLM calls, RAG systems, and tools working together. Okay, that sounds like a significantly broader scope than just deploying a single ML model in traditional MLOps. What are the key stages or components in this LLLF lifecycle that teams really need to master? Managing these complex, composed systems involves several critical parts. It naturally starts with data management and preparation, both for any fine-tuning you might do and for feeding your RG systems. Got it. Data first. Then, as we mentioned, prompt engineering and versioning, storing prompt templates, versioning them like code, running tests against different prompt versions.

20:06Then comes evaluation and quality assurance, which has to be a continuous process. You need to track performance against benchmark data sets to detect a model drift when the model's behavior changes over time, and often use A-B testing to compare different prompts or system versions in production. Continuous evaluation. Yeah. What about when things go wrong? Debugging these complite systems sounds like a nightmare. It can be, which is why monitoring and observability is perhaps one of the most critical and often overlooked components of LLMOs, especially for those distributed agentic systems. The key here is distributed tracing.

20:39Distributed tracing, like tracking a request across multiple services. Exactly, but tailored for LLM systems. It captures the entire execution path of a request. The initial prompt, the calls made to the ILM system, which documents were retrieved, any tool calls that were made, intermediate LLM reasoning steps, the final response. This lets you pinpoint the root cause when something fails or produces a weird result. It directly answers Ruiz's question, how do you debug them? This is how. That level of visibility seems essential. Okay, what's the final stage? Finally, there's deployment and governance.

21:15This includes the CICD automation we talked about, but also implementing crucial guardrails. Guardrails? For what? For security things like detecting and preventing prompt injection attacks or scanning outputs for sensitive data leakage. For compliance ensuring regulatory requirements like GDPR or HIPAA are met, maintaining audit logs. And also for cost control implementing budgets or rate limits. It really sounds like those orchestration frameworks we hear about, like Langchain, Lomindex, Semantic Kernel, must be pretty central to making all of this LMO-ed stuff actually work in practice. They're the glue holding these composed systems together, right?

21:49They absolutely are. Frameworks like these act as the essential glue. They provide the higher-level abstractions needed to build, manage, and reason about these complex chains and agents. They really embody Rui's principle of composition. They bring the software engineering structure that's needed to turn these ideas into maintainable applications. And this leads to a really profound shift, doesn't it? You said the deployable asset isn't just the model anymore. Exactly. The thing you deploy, the unit of change, is the entire system or workflow. It's like a directed graph of components, prompts, models, retrievers, tools, decision logic.

22:25Even a seemingly minor change to a prompt template is effectively a new deployment that needs to go through the whole testing and validation pipeline. Which perfectly explains why this new dedicated engineering discipline, LOMOs, isn't just, you know, nice to have. It's absolutely necessary if you want to do this seriously. Couldn't agree more. Organizations that keep thinking they're just deploying a model are going to really struggle to build reliable, maintainable and scalable AI applications. OK, brings us to Ruiz's fifth and final pillar, which is super pragmatic. He calls it crossing the production chasm.

23:01This is basically where most promising AI initiatives die, right? Because those flashy demos conveniently ignore all the messy real world constraints. That's exactly it. The demo works fine on a laptop with 10 users, but production is a different beast. So what are the biggest, maybe harshest economic realities that companies slam into when they try to move from that demo stage to actual production deployment? Two big ones often bite companies unexpectedly. Cost and latency. Let's talk cost management first. Unlike traditional software with fixed server costs, LLM costs are often highly variable and usage-based you pay per token, essentially.

23:39Which can get expensive fast if you're not careful. Very fast. So mitigation strategies become crucial. Things like intelligent caching of responses for common queries, implementing model-tiering routing, simple, low-stakes queries to smaller, cheaper models, and reserving the big, expensive ones for complex tasks. and adopting fine ops financial operations principles like close monitoring of usage and setting budgets or quotas to prevent runaway bills. Okay, managing the variable cost. What about latency? Yeah, latency optimization. Users, especially in interactive applications, expect near-instant responses.

24:14But, as we said, LLMs can take seconds to generate complex outputs. Which kills the user experience. It can, yeah. So you need mitigation here, too. We already mentioned asynchronous architectures to avoid blocking the whole application. But also using optimized inference engines, Ruiz mentions VLLM, which is a popular high-throughput serving engine specifically for LLMs, and techniques like quantization, which basically optimizes the model's calculations to run faster, often with a small tradeoff and accuracy, to speed up that crucial time-to-first token. So managing cost and speed are paramount.