We've all done RAG, now what?

0:03Welcome to practice.

0:30behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:48Welcome to another episode of the Practical AI podcast. I'm Daniel Whitenack. I am CEO at Prediction Guard and not joined by Chris today, but very happy to be joined by a longtime friend of the podcast and friend of all things AI and data science online, Rajiv Shah from Contextual AI, where he's the chief evangelist and also, of course, a TikToker extraordinaire and all things. So welcome back to the show, Rajiv. It's great to have you. It's great to be back. Excited to talk. Yeah, and we're close neighbors around the Midwest as well. I'm looking forward to seeing you at the Midwest AI Summit, which if listeners don't know, both Chris and I will be at the Midwest AI Summit, which is happening November 13th in Indianapolis.

1:42So I think as you put it on LinkedIn, Rajiv, if you live anywhere near Korn and you like AI, then this is the place to be. And actually, Rajiv will be there giving a great talk about kind of this idea of, I think you're going to be talking about this stat, the 95 % of pilots that fail, why that might be. Yeah, so great to have a fellow Midwest AI Silicon Prairie friend on the show as well. No, it's great to have these in-person events here in the Midwest. Usually you have to kind of fly out to some large city to get that. It's nice to be able to kind of get that right into Indy. Yeah, yeah. So if you're interested in that, go to MidwestAISummit.com and make sure and register for that.

2:29Hopefully, we'll see you there in person. It's going to be a fun time. But it has been a while since we've had you on the show, Rajiv, and I'm sure there's a lot to catch up on. It almost seems like last time we talked, there was kind of, we were all getting into RAG and thinking about, you know, agents and reasoning models. We're trying things. People were kind of gradually developing maybe what they thought would be a direction for kind of best practices or ways to approach problems. It would be interesting to hear kind of from then to now how you've seen that world advance and kind of these archetypal use cases that people are addressing with retrieval and related frameworks?

3:18Yeah, no, it's been amazing, right? About two and a half years ago, we were really giddy about the possibility of AI, but there was a lot of different things that we can do. And kind of since then, we've seen AI develop in certain ways where, for example, in code completion, code development tools, it's accelerated dramatically, even over the last year like that. And we're all kind of using those tools on a regular basis. And then in other areas, right, like chatbots, for example, we've seen some progress, but we've also seen that there's some things that are difficult to do with chatbots as well.

3:50But I think it's been incredible to see just the continued development of AI in terms of the capabilities, the smartness, the ability to kind of use tools to find search information that there's still so much for all of us to do in our lives, the kind of those of us that work with AI. Yeah, yeah. And I guess, like, certainly there's a lot of people still, I guess when I interact with customers or hear from other people in the industry, like still a lot of people are wanting as their first things to kind of whatever is build a rag chat bot, which maybe for those listeners out there that are maybe not as familiar with this, they've maybe heard, you know, rag this and raggedy rag that and all of that.

4:37Maybe just remind listeners what RAG is and how retrieval may fit into some of this AI stuff. Yeah. And I'll start simply. Even a couple of years ago, I was kind of working with the earliest large language models and people would be like, I like ChatGPT, but it doesn't know anything about me or my company. How can I train it to know all that knowledge? And what we figured out is that it's not about really training that model about the knowledge. it's instead finding that knowledge, searching that knowledge, passing that on to the model, and then be able to use that. And that's where kind of this idea of retrieval, finding stuff, and then augmenting the generation, augmenting the written response for that came about.

5:22And I think RAG today is kind of one of the most important, or it's one of the most widely used use cases, I'd say, in the generative AI space. Like every company is probably running some type of rag at this point for searching its internal knowledge, helping folks figure out through HR documentation, using it for customer support. There's a lot of use cases where we have huge amounts of information. We want to be able to find things from that, but we also want to use the smarts of AI for that. We don't want to Google search results of 20 things. We want a nice AI summary of it, or we want to extract all the information.

6:01I just want the names and dates out of all this. I don't need to read every one. And so that's where the AI really comes into it and why RAG is so powerful and we see it so widely used. Yeah. And I love what you were saying, just sort of contrasting this with training or even fine tuning. I find this, of course, to be a very widespread misconception about how these tools work. Even ChatGPT, the application, it does seem like it kind of, quote, trains on your your information that you put in your previous chat history, et cetera. And then like, it's almost like you have a model of your own, right?

6:41Which seems like it almost seems like OpenAI has a separate model for every person on the planet, which is not feasible from the data science training perspective. So could you highlight some of those things and just highlight, I know you educate a lot as well. So I'm sure you've seen this as well. There's kind of this jargon of training thrown around a lot, which is confusing maybe in how it's used. So yeah. Yeah. And I think partly is a lot of these companies want to make you think that they have like one thing that does everything in the world. But if you actually take a look at something like ChatGPT, it's a system.

7:21It actually composes of multiple parts. So if you were trying to build your own ChatGPT, these are some of the things that you would have to think about. So one is I need to be able to retrieve from lots of different sources. I have all my knowledge inside my companies inside of Confluence. I need to be able to access and retrieve that. That could be one thing. So one thing is using RAG, being able to access tools. Another thing is memory. I wanted to remember that last conversation I had, whether it was two things I said ago Or like one of the nice things about ChatGPT is you can come back a conversation or a day later and it remembers a lot about you.

7:58It may be a little bit scary sometimes how much it remembers about you and how well it can profile you. And so this is kind of what we see inside of AI engineering as context engineering, where there's a number of different parts of managing interactions with these models, whether it's RAG, whether it's remembering the memory, when it's knowing how to summarize conversations to do things like multi-turn so we can have those repeated conversations, but keep track of what was said earlier in our conversations as well. Yeah. So I love that idea of context engineering. How is that idea of context engineering kind of different from like what we would maybe from the data science, traditional data science side, consider like model training, I guess.

8:47So there's almost like a pipelining mindset versus an actual like GPU model training piece. Yeah. You know, I think what's remarkable nowadays is the amount of information that's stored inside of these large language models. I do a thing when I kind of talk about large language models where imagine if you were sitting outside and kind of reading a book and you could read all day, all night. If you read for 10 years, I think that's on the order of something like about a billion tokens. And these models are trained on the order of like 15 trillion tokens. So it's just an inconceivable amount of information that these models hold inside there.

9:29And I think one of the biggest improvements we've seen in these models is not only kind of stuffing all that information in there, is effectively using all of that information in there, that these models really today are untapped in terms of everything they can do. And it's one of these things on the technical side is why you see that the capabilities of the models have continued to grow without them necessarily having to grow immensely in size, simply because we're better tapping into all the abilities that they have inside them. Yeah. And I know you're up for this because we know each other, but I'll present a scenario to you that I sometimes comes up for us and, you know, whether I'm teaching a workshop or something and you can help us understand how this retrieval type of thing fits in.

10:18So, so I often, maybe I'm talking to folks in healthcare and what's interesting is there's all of this, you know, huge amount of data that these models have been trained on and there is good, let's say, medical information in that data, right? Or there's like care guidelines for nurses or medics or whatever it is. Let's take that as an example. The interesting thing is there's even like, if you think about factuality in a situation, it is kind of relative in the sense that if I'm a medic that works for this provider, let's say their care guidelines are different from another provider, or if like I'm treating, you know, maybe children, those guidelines are probably different than if I'm treating adults or senior adults or something like that.

11:09So there can be these facts out there that are actually conflict with one another. And it's really, you know, you talked about the context. It's really about the context that you're working in, what it should pull. So these models, assuming that all that information is public, right, like you said, it's been scraped off the Internet. All of that's kind of, you could imagine that maybe somehow that is embedded somewhere in the model, the base model. So whether we're talking about, you know, LAMA or GPT or whatever, how can, let's say I'm a healthcare company, but I'm interested now in my context.

11:45Now, how do I make that connection? So still use maybe I'm not going to retrain that model, but how do I use the model then in this context of kind of conflicting facts? Yeah, absolutely. As much as the models know and been trained so widely, they don't know everything. And so sometimes that's you have your own information. You want to pass that into the model. And so that's where that retrieval augmented generation comes in, where you want to grab that information and then we want to pass it into the model. And this is where, for folks in the space for a while, prompt engineering comes into play, where we think about the inputs that go into the model, where we take the facts that we know about our own given situation, along with maybe some instructions for what we want along with it.

12:30And all of that is manipulating the context, right? The larger world that this model is sitting within there. And by manipulating and giving it information, instructions, and other facts, we can get outputs that better match kind of what we're looking for like that. Yeah, yeah, that's, that's great. And I guess, you know, there's another term that's been thrown around the last couple years, which one would be this kind of retrieval piece, which you've talked about, and this idea that in the context of a certain question or something, I'm able to connect to one of my internal data sources, get that right information.

13:06there's this other thing that would be related to reasoning. Now, some people might kind of just consider that all kind of generative AI models reason in one way or another, but it is kind of now almost like a term of art. It means kind of a certain thing when we're talking about these models. Could you help highlight that also to put that in context? And then maybe we'll circle around and combine some of these things. Yeah. You know, a lot of times when we talk about models, We want to make it convenient to help people explain things. And so we kind of anthropomorphize them, right? We give them human-like qualities when they're not actually human-like qualities.

13:44And so, for example, reasoning is something we say for these models. Now, they're not reasoning like a human would through a problem. But what we typically mean kind of when we think about reasoning with these models is they're doing lots of extra steps. And they're doing these steps in a logical way to better solve a problem. So if you take kind of a math word problem, a great way to see this is if you take a word problem like the train is moving east at 50 miles an hour and another train is moving west another 40 miles an hour and you have to figure out like what point they're going to cross, right?

14:19There's no kind of quick answer to that. You need to kind of calculate the first train, calculate the second train, and then you can figure out the solution. And what we've done, and that's what these reasoning models have done, is we've trained them that don't come up with the answer right away. Think through it. Think about the first thing. Think about the second thing. Connect all the dots and then put an answer in. And the way we've done this is we've literally given the model. Well, we've trained it in multiple ways, but in some ways we've literally given the model an example of, hey, this is how I solved this word problem.

14:51I did this all these steps here. I want you to learn how to go through these problems step by step. And this is where we differ from two and a half years ago when the first models didn't know how to do that. Those first models, all we had trained it up was like, hey, tell me if a movie is a good movie or a bad movie, or tell me if this sentiment is happy or sad. But since then, we've had time to develop more training data. We've given them more complex training. And the cool thing is the models have picked up. They've been able to learn this capability for doing this. And so now we're able to do this much more complex kind of, I'm doing the air quotes of reasoning to solve these problems.

15:28Yeah, I guess one thing that people, and this is part of why I really enjoy watching your videos online, is you often break down a lot of this jargon and kind of help. It's almost like people feel the shock of all of this terminology and a new model coming out every 30 minutes or whatever it is, and just not really knowing how to deal with that. And there's kind of all of these things that have happened. So there's reasoning models, there's small language models, there's tool calling, there's retrieval, there's like all of these different kind of mix of things. Based on your experience, kind of, and, you know, both what you've implemented personally, also what you've seen and interacted with folks on, if people could kind of bring in some focus and maybe imagine a company's kind of getting into, you know, AI, however they view AI transformation within their organization.

16:29And they're thinking about some of the use cases that are kind of the initial ones on their roadmap at the time being, like, what would you encourage them to kind of like pick the signal out of the noise? So what would you encourage them to maybe focus on to see some of that time to value up front, not that they don't want to explore some of the other topics that are happening or read about them or whatever that is, but how would you at least recommend to get that best time to value or maybe just the things that are producing the most value? And you could maybe flip that as well and say, what are some of those things that are cool things, but maybe let's wait and see what happens.

17:14And maybe, maybe just sort of don't get distracted at the moment. Yeah. I mean, you're taking me away from all the full, all the fun, cool technologies that are the latest like automation things that I can kind of fire up like that. I think, you know, when you're thinking about this from the perspective of kind of a company, if you're kind of a manager in these situations, you have to figure out what use cases you want to put on a very different hat than just thinking about the technology itself. And I can tell you this, like I was burned from this, from personal experience when I was just starting out in data science, I was entranced by the latest technology.

17:51So I remember, right, we were talking about code development. Like 10 years ago, I was working at State Farm. And I think Andrew Karpathy has written his paper about the, I think like the unreasonable effectiveness of LSTMs. It was something like that. But part of that paper had the idea of you could translate code. You could complete code from one language to another. And I was like, hey, come on, guys. You got a lot of this code sitting around. Give me a few GPUs and some data and I'll solve this. Right. Totally naive. They didn't fund that project or anything. But I think, yes, it can be very easy to be kind of seduced by the technologies, kind of what a shiny demo is, versus when you're in an organization, you really have to think about kind of the problems that you have.

18:40Part of it will be, you know, how complicated is this from a technical point of view to get it up and running? That's one factor. But that's not the only factor. We also need to consider, like, what's the value to this organization? I talk to lots of enterprises on a regular basis, and I see often what I call science experiments, where teams like the latest technology, they go out and kind of run this stuff, but there's no way for them to actually get that implemented inside the company in a useful way. And they're literally just kind of interesting experiments that people are running like that.

19:14Does that get partially to like the 95 % of AI pilot failing type of report from MIT? Absolutely. Now, the 95%, of course, is like a little bit of a hype number that they like to put out in this. And for those of us who've been in the space for a while, we remember the fact of 80 % of data science projects fail, I think was something that we had. And to some extent, that's okay. You can't expect every initiative, every experiment, everything that you start to succeed. You want things to fail because partly is if something works, that means you have to maintain it. You have to monitor it. You have to put up a lot of guards around it.

19:55There's a cost for something that actually succeeds. But when we talk about AI, it's very easy to build a cool demo, but it's not only the value to the company. You have to figure out how to integrate this into people's everyday work life. And so you can build a very shiny widget that sits and can do something awesome. But if that's not inside somebody's regular workflow of how they work, the tools that they work in, if it's not, if they're not properly trained on how to use it, if their leadership isn't supporting you to use it, like there's lots of factors like that that go into why people might not actually adopt and use a technology.

20:33And it's really nothing about AI. It's really about organizational change and introducing technology into companies. Yeah, I know from just a founder perspective, it can be, you know, just from a different side of this, it can be frustrating when you see like, oh, there's this, you know, company over here that the technology side is fairly simple. Like, oh, it's just a simple model that does a simple thing or a browser extension that does this. And you're like, wow, you know, like I could have vibe coded that in a weekend. How are they scaling to the moon? And we have all this cool technology.

21:13And I think part of it is that side of it that you talked about. Part of the hard problem is cracking what actually does provide value to your organization, what can be adopted, how you communicate that, how you tell that story, and how you deliver on your promises, how you provide customer support. A lot of that is not really related to the technology and that component. So maybe, I don't know if this would be accurate to say, but maybe the first step people could take is just getting something off the ground that's fairly simple and interacts with an AI model. Maybe it's just to do a very simple task, but really pushing that through, like you say, to be embedded in a product or be embedded in a process, that may be the best way for people to kind of start that journey is to, yeah, really start from that simple side and deal with some of, in all honesty, the harder problems around the periphery of the technology.

22:19And, you know, I think a big part of that, like what you're saying, is just to get closer to those end users, the stakeholders, because I think once you often cut through that, sometimes you figure out that really they don't necessarily need a fancy kind of GPT-5 model to solve their problems. Maybe you can solve it with almost a simple if-then rule that you can just implement in some place. And so this is where kind of looking at the data, spending time talking to those end users like that often gives you a much better result, right? It's going to give you the biggest bang for your buck than going out and reading some archive paper.

22:57Yeah, yeah, that's true. And I guess now you should just have the AI model read the paper for you and give you some summary points. I'm a big fan of that. On my walks, I often sit and I'll talk to chat GPT and we'll talk through papers and what are the main technical points and stuff. So yeah, I'm glad I'm not the only one. So thanks for validating that. I guess getting back like more into the development retrieval kind of reasoning stuff. What are some of that now that we've been working with this technology for a while, we've got more cycles like we someone can spin up a rag pipeline and, you know, whatever, 10 minutes I can spin it up and I have something going.

23:39But it's another thing to kind of, of course, scale that, maintain it over time, deal with some of the issues. You know, what I guess my question is, what pitfalls are people falling into that maybe we didn't know about whatever it was a year ago when we were kind of just getting into these initial kind of naive rag sorts of things? What what challenges are consistent challenges and pitfalls do you see people kind of falling into? Yeah, I think the consistent thing I see with with something like rag is it's fairly easy to build a quick demo. You can grab an off-the-shelf embedding model to do that.

24:15You can combine that with a generation model like an OpenAI model, and you can build yourself a quick kind of proof of concepts. I think the trouble that people get into with it is scaling it up. It's great on 100 documents, but now all of a sudden, I have to go to 100 ,000 or a million documents. How am I going to do that? Or, you know, when I first did my demo, I did a couple of very simple queries, text extraction queries. But now when I put it out in front of my users, I find out all of a sudden they're not giving nice one sentence queries. They're just asking two words and then I need to add a query reformulation step or something to do that.

24:57Or the accuracy is not kind of what I was looking for. And so I've added a bunch of pieces in there. I've added a re-ranker. I've added other steps. But now my latency is suffering, right? There's all these kind of trade-offs as you kind of get to production. And then you're like, oh, you know, do I go back and you go and you look online and you see that, oh, wait a minute, there's like 15, I think there's like 25, 30 flavors of rag. And you're like, oh, did I set up my infrastructure wrong? Or, oh, do I go back and I change my chunking strategies? I can see, you know, there's 10 different chunking strategies people are doing that.

25:31And so I think this is the cycle of where it's very easy to get started, but getting to that final kind of production quality rag can kind of be a little worrisome. Yeah. And do you think that that's where the real human engineering piece of this development still will be with us for some time? Because to some degree, you can describe, let's say I describe that sort of problem to my AI coding assistant, is it reasonable for me to think that like that kind of debugging and process could help me or that kind of assistant type of thing could help me get to the bottom of this or update my retrieval pipeline and that sort of thing?

26:13So I'm kind of optimistic that the reasoning models that we have now are going to get us much farther towards helping you solve that problem. Now, of course, that reasoning model has got to understand like how you're thinking about how to solve that problem. But already today, if you take the traditional RAG approach, for example, but you pair it with one of the reasoning models that can make tool calls, that can kind of look at the results that come back, think about it, decide, hey, I want to requery it in a different way. You can improve the quality of your results in that way by kind of using that reasoning to do that.

26:48So I'm pretty optimistic that we're going to keep finding new ways, as long as there are workflows that we can train these models on that are kind of fairly, fairly, let's say, logical. Archetypal. Yeah, exactly. Gotcha. Like a way we can kind of connect the dots and teach the models to do that. That I think a lot of these things that if we have the, if we give it the budget for spending time thinking, doing those tokens there, it's going to cost us more latency, but we're going to see better results. And I think some of us, we already see that in using some of these tools like deep research, where we can see that by spending more time on the task, it's able to give us a better result.