1017: Vector Search, Agentic Memory and Effective RAG, with MongoDB’s Pete Johnson

23:34Jon Krohn:They now can be thinking about, okay, what is my, what is the client of my business want? Or what would make the experience of this product better for a user? And now every individual can be way more impactful than ever before. And I've made the case that I think just like bank tellers, I think it means that an organization when they're like, wow, look at how much value I'm getting from this data science hire. Look at how much value I'm getting from the software development hire. How many more of these can we get? If software engineering is going to disappear as a profession, then why is Anthropic hiring 50 of them?

24:07And why are they offering, you know, 570 K a year comp for it? To get their keystrokes.

24:14Jon Krohn:Yeah. Oh man. Yeah. All right. So this has been great for digging into your kind of general philosophy on where the market is. Let's zoom in a little bit more on what you're doing at MongoDB. Sure. You guys are specialists in databases, obviously. The big thing historically at MongoDB, Like when I think about the word MongoDB for me in kind of a vector space map in my own brain, a lot of the neurons that go off when I hear the word MongoDB is NoSQL. Sure. So probably a lot of our listeners already know what NoSQL is. But just as a quick primer, maybe you can tell us how NoSQL is different from traditional structured databases.

24:58Jon Krohn:Sure. So if you look at the history of databases, the kind of triggering white paper was written in June of 1970. That happens to be the year I was born by a guy named E.F. Codd, who was an IBM researcher. and he gave rise to this notion of what we now know as SQL and how do you organize data on disk. And at the time, if you think about the issue that developers in our early stages as a profession were having at the time was the cost of storage relative to, say, compute and memory. We hadn't yet had Moore's Law kick in over 50 years like we have now. So you had to be very precise about how you laid out data on disk.

25:46And that's why we see things like, you know, my wife and I share a mailing address. And if we both have an account with a retailer, let's say, you store the address once, you store customer records for each of us separately. and when you need to get a name and address of someone, you do what's called a join, right? Where you do a search on one table to get the address, you do a search on the other table to get the name, you sometimes have a third table that connects the two. You're making three reads on disk for that, but the storage on disk is very efficient because you're storing the least amount of information possible.

26:24This is the foundation of SQL and it's what everybody has been using and what gets taught in universities and coding schools as the right way to do it. So the term for minimizing that amount of data on disk is called normalization. And there's use cases for which it actually makes sense to denormalize. So where normalization will give you efficiency of how you store on disk, what I just walked through, you have to do two or three reads on disk in order to get that piece of information? What if you denormalized that? What if you think about it as, okay, disk actually isn't as expensive in 2026 as it was in 1970.

27:08So in that retail example, suppose you have my address and my name in one record, and you have my wife's name and her in the same address and a second record, I'm duplicating the address in both places. But from a performance perspective, I only have to do one read from disk so it's faster. Now, the way that we nuance that with MongoDB is when we store that data, we're storing it in a version of JSON so that when it gets serialized off of disk and we ultimately send it back to you from the API call, it's already in a format that your language of choice can immediately use. And that also adds to the fast benefits.

27:51So like when do milliseconds matter for you versus when does disks, you know, the cost of disk matter to you? So it's like the scarce resource in 1970 was the cost of disk, but the scarce resource in 2026 is time.

28:08Jon Krohn:Great explanation. I love all the kind of historical background we're getting on pretty much any explanation that you provide today, Pete. You got an old bald man on the show today, John. And you've got not quite as old, but getting there and very bald, even balder man interviewing you. For people who, oh, I guess you can't really tell because of my head. I was going to say for people who are only listening to this as opposed to viewing it. But anyway, so at MongoDB, one of the places where you've been able to make a lot of impact in the AI world that we're in now is around vector search. Yes.

28:47Jon Krohn:And you have framed vector search as the new frontier, and you work directly with enterprise teams on the architectural choices behind production AI, including RAG, retrieval augmented generation patterns, embeddings, vector search, retrieval quality and evaluation. Your perspective bridges the technical details of these systems with the practical realities of making them reliable at scale. So there's, I mean, we're probably going to end up digging into this question, like the statement I just said for the whole rest of the episode, because there's so many cutting edge technologies that depend on great databases.

29:26Jon Krohn:But let's start with Vector Search. Why is Vector Search the new frontier? So it's the new frontier because of its importance within an application architecture that makes use of some generative LLM. So if you think about how this has matured, when ChatGPT came to market in 2022, the Its application architecture is you take a query or a question and you put it in the context window of the LLM, the LLM does its processing and gives you some very smart looking natural language response. So by the time we get to like spring of 23, you see headlines like ChatGPT passes the bar exam. The LLMs have two fundamental restrictions in them that they still have today.

30:11So number one, they're trained on public data, not proprietary data. So if I'm trying to solve a business problem for my business, how do I inject my content into the LLM without having to incur an expensive retraining or fine-tuning process? How do I do that? Well, in 2024, people started to use retrieval augmented generation for that. The other limitation that you have is that the LLMs have a knowledge cutoff. There's a point at which the LLM vendors have to stop training and start to inference the models. And by their very nature, the LLMs don't know anything that happened after that. So in the spring of 23, if you had asked ChatGPT, what is MongoDB stock price today?

30:54It can't tell you because that's after today is after the knowledge cut off. In 2025, we started to see the MCP tools and tooling take advantage so you can make live calls out to places. And now you have the basics of what current agentic architectures look like, where you've got some RAG pipeline, you've got some set of tools, you've got the generative LLM. You now take the results of one pass of the LLM and it becomes the query for the next pass. So you have this loop. And then you have agentic memory that keeps track of things across sessions within that loop. And those are kind of the basic components of an application architecture when you're talking about building agents now.

31:37And vector search tends to be a big part of those RAG pipelines and of those agentic memory. And we can get into some of the details about how MongoDB implements that and why things like retrieval quality are important, why things like scale are important, and how things like our acquisition of Voyage AI help with some of that.

31:59Jon Krohn:Agents are getting smarter every day, but even the smartest agents get stuck without the right context and the right tools. That's where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now their new developer platform is turning that workspace into infrastructure developers can build on. The piece I keep coming back to is how easy it is to ship something real. The CLI authenticates in one line, workers deploy without provisioning any infrastructure, you write your code, deploy, and you're done.

32:30Jon Krohn:For me, that unlocks building purpose-built tools for my custom agents with the predictability and custom logic I need. Think a guest prep agent that pulls a researcher's papers, recent talks, and citation graph on demand. Tools my agents can actually call with parallelism and predictable behavior, not just hope for. Learn more about Notion's developer platform today at notion.com slash superdata. That's all lowercase letters, notion.com slash superdata to try Notion's developer platform today. And when you use our link, you're supporting our show, notion.com slash superdata. For sure. Let's definitely get into that.

33:07Jon Krohn:So for applications like RAG, where we allow a system to be able to search over effectively an infinite number of documents, when we're pulling out what those relevant documents are, we're using vector search, which you were just talking about as the new frontier. what common mistakes do you see teams make when they try to optimize RAG, Retriever Log Metageneration, for enterprise-grade reliability? So here's where the details matter. So the way that things started to change in 2024 when we started to introduce RAG into this ecosystem, you take some documents, you pass them through an embedding model, and you then store them in a vector database.

33:51That's the data ingestion part of RAG. Then at query time, instead of taking that query, the question, and putting it directly into the context window of the LLM, you instead take a side quest. You take that query and you pass it through the same embedding model that you used before. You then can do a vector search, or as it's sometimes called a similarity search, and you get related documents that are related to the query that you put through the data ingestion phase. You then take those documents and along with the query, you put them into the context window. Now the LLM knows things about your proprietary information in a way that doesn't require fine tuning or doesn't require a retraining.

34:33And so that's how you get around that proprietary limitation of the LLMs that are trained on public data. The biggest mistake I see people making is assuming that that embedding model is commoditized. It's not. But the basis for what you pick for that embedding model will have a direct impact on the quality of the retrieval you make when you try to do the semantic search with your vectorized query and get the similarity documents. This is why we purchased Voyage AI in February of 2025, is we saw this as a differentiating feature in the marketplace. There's a benchmark that all the embedding models use that's on Hugging Face, it's called Arteb.

35:16And depending upon the model, we score as much as 14 % higher than some of the embedding model competitors. Most people choose an embedding model today based on what's convenient for them, based on their cloud of choice. What I would say the biggest mistake is not looking around and seeing if you could get better retrieval quality out of your rag pipeline, what are some problems you could go tackle? If you could do 14 % better, are there problems that you could solve that you aren't able to solve today with whatever your default embedding model might be?

35:51Jon Krohn:I didn't know about RTEB before, but I'll have a link to it in the show notes. It stands for Retrieval Embedding Benchmark. And yeah, Hugging Face published it in October of last year. It sounds like a really useful benchmark for retrieval, for vector search. And so I will have that for all of our listeners. So in the past, you've laid out quality, storage efficiency, and developer friction as the three criteria for good vector search. Do you want to tell us a bit more about those three criteria? Sure. So let's start with the last one, the developer friction. More people will build agents in the next three years than have built them in the last three years.

36:34In order for a broader set of software engineers to take this on, you need to make it easier to work with. So reducing that developer friction. And I'll give you a simple example. One of the things that you have to do when you pass that data through that embedding model is how big is my vector space going to be? Anybody who's taken algebra understands what 2D is and probably what 3D is. When you create an embedding space, you typically, you might choose 1024, 512, 256 as the number of dimensions. And while I can't think in 256 dimensions, the embedding models can. What's the right number of dimensions for you to choose for your use case?

37:19And the truth of the matter is a developer has to iterate over them. So how can you remove friction from that iteration? You pass your corpus of data through the embedding model, say at 1024 dimensions, you run some tests and see what kind of retrieval quality you get. You then take your corpus of data, you run it through again at 512, and then you see what kind of performance you get. And it's this balance between retrieval quality and storage space, both on disk and in the index in memory, that a developer has to constantly iterate over. Every other embedding model in the market would force you to re-embed your corpus of data every time you want to try a different number of dimensions.

38:06But the Voyage models have a feature called Matryska reasoning, which the name comes from Russian nesting dolls. If you think about how a Russian nesting doll works, you've got the big one, you take it apart, and inside you've got the same thing, but at a smaller dimension. When the Voyage models generate the dimensions, they're ordered. So what that means is if you want to test from 1024 to 512, you just lop off the last 512 of what you already generated at 1024 and you can now run your tests again. You don't have to go through the mechanics of re-embedding your entire corpus of data a second time at 512.

38:43So it reduces the amount of time it takes you to test that iteration in a way that nobody else on the market has. So that's one example. There are several others that we have based into the, that's baked into the feature set. Things like shared embedding spaces where different sizes of the models are compatible with one another. You can very quickly switch between them for different use cases. There's something called auto embeddings where you don't have to manage the maintenance of your pipeline on your own. We'll do it for you. You tell us what field in your JSON you want to embed based on, you tell us how many dimensions, you tell us which Voyage model, we'll take care of the updating of both the embeddings and the index as that data either changes or you introduce new content into that collection.

39:31So that's all what I mean by the developer friction and these are all things that MongoDB has built into, not just the vector search, but this better together Voyage embedding models and re-rankers plus the core MongoDB vector search.

39:47Jon Krohn:I love that answer. Just as I have loved actually all of your answers in this episode, you do such a great job explaining every technical topic that you go into. You are really experienced at explaining vector spaces and everything related to it. So thank you for that. Something that's exciting with you mentioning some of the MongoDB functionality that is designed to reduce developer friction as they use vector search standalone or as part of RAG or whatever, something else that mongo recently announced is new shared embedding spaces what does that mean so most embedding model vendors offer two to three sizes of the embedding models and they come at different price points and with different base retrieval quality so there's usually like a small medium and a large what we recently introduced is what we're calling the nano which is an open weight version of the series four family of Voyage models.

40:40And that one's free. That one's out on Hugging Face. And if you've got the right CPU on your laptop, you can run it there. Shared embedding spaces is the idea is that embeddings generated by one of those four models is compatible with all of the others. So what that sets up is I just talked about, okay, suppose for my development cycle, I take my corpus of data and I embed it with the large, let's say, and those are the vectors that I store in my vector database. But as I'm a developer who's trying to figure out what the logic of my agent should look like, maybe you don't want to incur any token costs.

41:18So maybe what I do is I download and I run the nano locally, I embed my queries with the nano, but I can now do searches against the data that was embedded with the large. You take a little bit of a retrieval quality hit when you're using two different models, but the embeddings between those four models are all compatible with one another so that you can eliminate token costs during your development cycle if you want.

41:44Jon Krohn:That's cool. That is a novel concept for me. I haven't seen that before. First one in the market to do it. Wow. Nice. That sounds really useful. You mentioned a few minutes ago, a term that I've been chomping at the bit to get back to, which is agentic memory and how that interacts with all of these kinds, you know, when we're building a production AI system today, getting the agentic memory right is critical to having it work. And it seems like you're going to have a lot to say with respect to that. Yeah, some of the advancements we've seen in agentic memory here, the first six months of 2026 have been in direct response to this token maxing part of the conversation that we just had.

42:32So in its early days, agentic memory was just real simple short term, long term, where short term was keeping track of all the responses that happened in this session. And long term was keeping track of responses that happened over multiple sessions. And anybody who's used any of the modern chatbots, which are slowly morphing their way into being agents. So whether you're a chat GPT person, a Gemini person, or a Claude person, you've all experienced this where maybe a year ago it wouldn't remember anything outside of a session, but we got long-term memory and now all of a sudden it can remember things across sessions.

43:07So we think that there's something better out there than just short-term and long-term memory. So let me paint a case for you. So suppose Suppose a question comes in and my agent passes it to the question to the generative LLM and it generates a response. What if I can then save that response in a more sophisticated agentic memory so that when the next time someone asks a similar question, instead of automatically taking that question and sending it to the LLM and incurring all of the token costs that are associated with that. What if instead I could do a vector search into my more sophisticated memory so that I could get candidate answers and then I could use a lower cost evaluative LLM, something like Quinn or Llama.

43:57Is this answer good enough to answer this question? If it is, you return it and you don't go to the generative LLM. If it's not, then you pass the question on to the generative LLM and you go through the cycle again. So what we're starting to see with these more sophisticated memory types is it's a way to combat token maxing. And it also has the nice side effect that you get consistency out of it. We've all experienced something where if you ask a chat bot a question and then two weeks later you ask it the same question, because the underlying LLMs are probabilistic, you get two different answers.

44:35Well, in a business situation, that can be catastrophic. You need some consistency out of the answers. And if you generate the answer once, keep track of it in memory, able to pull just that answer and not just sort of less sophisticated short-term and long-term sessions, but a specific answer. If you can pull that with better retrieval quality on your vector search out of that agentic memory, now you can use that to avoid the call altogether and save on your long-term token costs.

45:04Jon Krohn:I love that. You mentioned at the end there, the importance of designing these systems effectively for commercial use cases. And Mongo has obviously been developing a lot of products, optimizing in a way to allow organizations, kind of the bigger they scale, the more efficiencies, the more value you're going to get from MongoDB. So you have, in the past, you've connected vector search, embeddings, and MCP to the idea that databases must store what's accessed all together. extending a decades old denormalization principle that you've been talking about kind of right at the beginning of the episode into AI retrieval design.

45:45Jon Krohn:Can you like pull these, pull those different concepts together into what MongoDB is delivering for its customers? So the root of what you're asking is how MongoDB vector search works. So let me get into those details for a little bit. And then I'm gonna make a very old man statement that is either going to make you smile or you're going to have no idea what I'm talking about. So the way that this works, so like I said, within the core product, MongoDB stores things as JSON structures. So we call them documents and we put them in collections. So we don't have tables and rows, we have documents and collections.

46:22Each one of those documents is a JSON structure and you get all the flexibility that you would imagine with a JSON structure. And as an extension of that, what is a vector? A vector is just an array of floats. It's an array of floats that was generated whether you pass an image, a movie, a piece of text to an embedding model. Ultimately, what you get back is an array of floats. So for us, that's just an additional attribute to the documents you already have. So if I just add one more attribute to the document in my collection that's now an array of floats, I can now generate a vector index on that.

47:00and that's how the search works. But I have a choice here. I could choose to use data that's already in my documents. So suppose I have a book. I have a document that describes a book and there's an author name, there's a title, there's number of pages, there's a year published, there's a synopsis, and there is a text field that is a URL to a JPEG for the cover. In the example that I just walked through, you could take the synopsis and you could generate vectors based on the synopsis. And that's an example of when I do the query, my similarity search, my vector search, what I get back is some scores and I'll have the document that has the actual piece of data in it.

47:45And that's how most people do it. What I would make a parallel to is in old school C language programming, you had a choice when you created a function or a method where you could pass by value or you could pass by reference. And what I would argue is if you've got the data in the document with the vectors, that's like passed by value. But what about that cover? I don't have the JPEG in the document. I have a pointer to where the JPEG is. I could take that JPEG, I could run it through a multimodal embedding model, and it's still going to generate a list of floats on top of which I I could generate a vector index.

48:25When I do that search in that example, what I get back is a string that represents a pointer, a reference to where I can go find the JPEG. So it's not like the synopsis example where I have the data, now I can go about my way. But now I have a reference where I make a second call to go grab it from its system of record. But that gives me choice. I could choose to pass by value and have the data there in the document, or I could choose to pass by reference because I don't want to have to copy that data over in this example to MongoDB. Maybe I want to keep it in its system of record that there's reasons why I might want to do that.

49:00And I'm willing to take the runtime hit of making the second call to go fetch it once I get the vector search back. So it's giving people that level of design choice. And in that example, I just gave not only could you do a vector search, you could also do a lexical search. You could do a hybrid search. You could pre-filter based on the author name or the year of publish. There's all kinds of mechanics we offer to make the nuance of how you're doing that retrieval far more powerful than you have with some of the alternatives. And it's all based on that same document model. So because it's based on that same document model, you get all the replication, all of the sharding, and all the additional security features that we already have baked into the core product.

49:45You kind of get that for free with the vector search.

49:49Jon Krohn:On this podcast, I'm always going on about how Claude Code is mind-blowing, but now Claude Cowork is making my jaw drop as well. For example, I recently wanted to quantify how healthy my sales pipeline is for my AI consulting business. I simply asked Claude to estimate my sales for the coming quarter, and it brought info from relevant Google Sheets and my Gmail to create a professional spreadsheet of clients with estimated revenue for each one. Whoa, this might have taken me a day. Instead, it was done flawlessly with Claude Cowork in minutes. Claude is the AI for minds that don't stop at good enough.

50:20Jon Krohn:It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. Ah, and you'll appreciate that I can ask Cowork to show me data, such as my sales spreadsheet, and it provides an interactive chart right in the conversation. For problems worth solving, get started with Claude at Claude.ai slash superdata. That's Claude.ai slash superdata. and check out Claude Pro, which includes access to all of the features mentioned in today's episode.

50:49Claude.ai slash superdata.

50:53Jon Krohn:I love it. It does seem like a pretty obvious choice for a data backbone. If you're going to be building AI applications, Gen AI, Agentic AI, it makes a lot of sense to use Mongo as your data backbone for sure. It's something that we've been using in startups, tech startups that I've had for over a decade now. And it still continues to be the default choice for us. Yeah, so much flexibility. It's easy to get started and it scales, you know, it scales with you longer term so that when you run into things like GDPR or you need to start getting data closer to users for different things, that's all baked into the core product.

51:32Jon Krohn:Awesome. Well, Pete, as I already said earlier in the episode, I have thoroughly enjoyed listening to everything that you've had to say in this episode. you were so slick at explaining technical concepts that you might think a diagram would be needed to explain it. But in a podcast, we don't have the luxury of diagrams. So we have to rely on outstanding verbal representations of the ideas. And you absolutely nailed it. Really appreciate it. You are very kind. It's almost like I was raised by teachers. Yeah. And have decades of experience in databases and AI. it's all coming together really nicely for us uh yeah so thank you so much for this great episode before i let you go i have two final questions that i ask all of my guests the penultimate one is do you have a book recommendation for us the only book i've read four times is michael creighton's sphere the movie is terrible don't watch the movie but the book is you know at the height of his pre-jurassic park powers um and you know involves computers and time travel and And, you know, what else could someone like me ask for?

52:42Jon Krohn:I love it. Oh, the time travel paradoxes that come up. Oh, I watched two time travel films or two films that involve time travel as key plot points over the most recent weekend. And it's hard to do that without some kind of issue coming up. But maybe Michael Crayton can do it. He does it well. And now I'm wondering whether you watched, you know, did you watch Avengers Endgame, Back to the Future, Austin Powers or Hot Tub Time Machine? I actually none of the above. Amazingly, I watched Star Trek, the 2009 version. So that was the first one directed by J.J. Abrams. And yeah, time travel is a big part of that episode.

53:28Jon Krohn:And then on Sunday, I watched a comedy that I can highly recommend. I actually watched it for the second time in about a month. it's nirvana the band the show the movie that's the name of it well i can never get enough of leonard nimoy so you got me with this 2009 star trek nice there you go yeah you get you get two leonard nimoy's well you get two spots one leonard nimoy there's only one leonard nimoy that's right that's right that must have been one of his final films yeah i think it was final Final question that I ask before I let guests go is how can people follow you or follow MongoDB after this episode?

54:05Jon Krohn:What are your recommended places for that? Oh, sure. So both me personally and the company in general, very active on LinkedIn. So it's easy to follow us there. If you're looking for more company information, it's, you know, MongoDB.com or we do a lot on the YouTube channel as well. For sure. Great. We'll have links to all of those things and anything else we talked about in this episode in the show notes. Thanks, Pete. Again, I hope to have you on the show. sometime again soon because I felt genuinely like we could have talked for hours. Well, that's better than if you couldn't wait to get rid of me.

54:36Thank you so much for the time. I really appreciate you and the time of your audience.

54:40Jon Krohn:I loved having Pete Johnson on the episode today, on the show today. I hope you enjoyed it as much as I did in today's episode. Pete covered why the first agents delivering real ROI in production are employee facing with a human in the loop, since firms already know exactly how they measure an employee's performance. So any jump in those metrics can be attributed directly to the AI. He talked about why the embedding model is the most underrated choice in a RAG retrieval augmented generation pipeline. He said that most teams default to whatever their cloud provider offers, while models like MongoDB's Voyage family score up to 14 % higher on retrieval benchmarks.

55:20Jon Krohn:He talked about Matryoshka embeddings named after Russian nesting dolls, which let developers test smaller vector dimensions by simply lopping off the end of larger ones instead of re-embedding their entire corpus. And he talked about his bank teller versus toll booth framing of Javon's paradox, where teller jobs grew after ATMs because the role had somewhere more valuable to go. And the question to ask about any job facing AI is whether it has the same room to move up. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Pete's social media profiles, as well as my own at superdatascience.com slash 1017.

56:02Jon Krohn:Yes, this was episode number 1017. That's where that number comes from. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, partnerships manager, Natalie Zajski, researcher, Serge Masise, and our founder, Kirill Arimenko. Thanks to all of them for producing another outstanding episode for us today for enabling that super team to create this free podcast for you. We are deeply grateful to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. And if you know someone that would like to know how they can be getting a better ROI from AI projects or to be able to understand better how NoSQL or MongoDB style databases can be useful, in the agentic AI era, then share this episode with them.

56:51Jon Krohn:Review this episode on your favorite podcasting platform or on YouTube. If you write an Apple podcast review, there's nothing more valuable to us in terms of an action that you can do. So please consider doing that. And yeah, subscribe if you're not already a subscriber. But most importantly, I just hope you'll keep on tuning in. I'm so grateful to have you listening. And I hope I can continue to make episodes you love for years and years to come. Till next time, keep on rocking it out there. and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.