Production-Grade AI Systems with Fred Roma

25:27That's not about giving anything to the LLM. It's about your application being able, with very good information retrieval, to say, okay, my user is asking me again for what is my credit limits on my credit card. A lot of things can be handled by the LLM in terms of how to answer to a user asking question. But if I really want this information, I also have to go and find my private information of my bank about what is a real limit. And I will provide this information to my user combined, but the LLM will never see this private information. So maybe just to clarify the overall pattern before going into the security details, that's super important.

26:03I think that is very important in terms of what sets of data are, I think the term I sometimes use is moderated through the LLM. So even if the LLM, the LLM may be the UX delivery, but do I load this data, pass it to my model and have it presented? Or do I sidestep around the LLM because this has to be right and I can't count on it not hallucinating something about it? Yeah, yeah, absolutely. And there are different patterns there. Most of the time, what customers will do is, again, if a user, an agent, want to do something that does require information retrieval, you will first look for the information that will be relevant.

26:42I'll stick with the same example. What is the internal document that explains what are the limits on credit cards? And then we'll insert that in the prompt of the LLM. So, yes, that will go through the LLM from the prompt and the answer of the LLM, but it's not stored anywhere on the LLM side. It's not been part of the training of the LLM. Sure, yeah. And if you decide, and then that's up to you as a customer, where do you want this LLM to be hosted and where do you want these queries and this token to be served? And so you can decide depending on your security sensitivities. Many customers say, okay, I'm totally fine with having my LLM on AWS or OpenAI or Azure, et cetera.

27:20Or some customer will say, no, I want to do that, but I want some specific security agreement with these providers to make sure that my data is never shared. And some customers say, you know what, I want to host my own LLM. You can do that if you want. What is important is that there is nowhere the blending, if you want, of how the LLM is trained and your private information. Yeah. So coming back, though, to building applications with these pieces, I think I'm curious to understand how you're seeing people defining these lines or barriers. Is it changing at all in terms of how you're managing security at the data layer?

27:58I mean, security has always been a very, I mean, customers trust us with our data as a data platform. So it's always been top of mind anyhow. But I would say that with this AI-specific application, we see more and more, I would say at least even more discussions about this LLM integration, the one you spoke about before. And one of the big value of MongoDB is you can run it anywhere. And when you say run anywhere, sometimes people tell us, oh, you are cloud agnostic. You can run it on AWS and GCP and Azure and etc. Yes, that's true. We can also run it on premise. You can also, we have an enterprise and you can do that in your own data center.

28:37And we see customers that are doing that. What is very interesting, I think, in this AI world, we see customers that are saying, you know what, for this use case, I'm totally fine if it's in the cloud. But for this use case, I really want to make sure that my data is never in any cloud provider and never touched by any LLM provider either. And so I will run it on-prem. And you start to have this kind of... And again, I think it's so early and things may evolve, et cetera. But the fact that they have a choice and they can decide what to rely on as a cloud provider, as their own data center, I think is pretty interesting.

29:11And I do believe it's more and more a discussion topic. That's fascinating. and when you're seeing that are you seeing them doing this within the context of the same application so you're having to kind of have security boundaries and federation and however that's working or like yeah how does that work i'm not sure i can extract one pattern one one answer to that you have customers that will say okay i will really have all of my data will be on-prem and then some application will be in the cloud and some won't i see some customers saying you know what, I want my data to be in my data center. So I want MongoDB Enterprise Advance, and that's how I manage it.

29:47But I'm still okay to do a call to an LLM outside of my boundaries, because they believe that they can control, and they are right in many cases, they can control the prompt. So it's okay to send some information as soon as it's something that your application control, but at least nobody's seeing the raw data. So I really see different kind of patterns there. And I wouldn't be able to tell which one wins. I think what is very important though, is that overall, I think there is really this intent to remain as flexible as possible. I think, say, you know, I'm back to the previous point. Maybe this LLM is good for me right now, but in six months, that will be another one.

30:22Maybe this cloud hosting is good for me right now, but then I will want to go on-prem for any regulation or specific concern later on. I think there is really a willingness to remain as flexible as possible and to have options on the table, I would say. That gets to kind of a somewhat different topic, but you know i think one of the things that these tools are doing is they're changing the speed at which people are operating they're changing absolutely kind of how fast we're moving they're changing how adaptable we need to be how are you both internally and with your customers like rethinking the way that we organize the teams doing this work oh yeah yes that's that's uh okay i mean there is a product angle i thought we were initially going to the product angle and and uh we just go there super uh super quickly but i want to go to the team organization i think that's super important as well.

31:10But from a product, I'll go back to if you want to go fast, but you have to move your data around and to... I mean, any of us who have been software developers or architects at one point knows that when you have to optimize for latency, performance, iterate quickly with network layers and data to transfer, it's just a nightmare. So we say that that's one of the key arguments, not just for MongoDB. I think that's a big value of MongoDB, But overall, I think the database vector search and search are the same place. And you don't have to do this ETL as this kind of difficult network configuration.

31:44I think that's a big one for your question as well about the speed of development and iterating. Now, if your question is going more to the team organization, I think that's a very good question as well. Even so, I mean, two things. I think what's happening right now is that it can be also very thrilling when you are a product manager or business person say, oh, I can build it myself. I can beat fast. And personally, there's something I love about that. What I love about that is that instead of trying to debate about maybe some text and some PowerPoint, you can really show. I love that. I think the risk is to believe that it's easy.

32:19Yes, it's easy to show something. That's the old engineering and product manager. Sometimes they have to align on that. I think it's great if you use that well. If you use that as a way to say, oh, and just put it to production. Well, for some stuff, yes. But how will you scale? And because that's all of these tools, right? They're making the code, writing code super fast. Well, reviewing code is not faster. And making your security assessment of this code is not faster. And defining the right architecture is not faster yet. So I think it can be awesome if you know what it is. It can be a bit dangerous if you believe that, oh, then that's it.

32:54And I can just push it. So I don't know if I maybe pivoted a little bit to your question, but that just made me think of this point. No, I think it is key. and it kind of gets to a couple of different questions, some of them related to the product piece and some not. So like one of these things is going from zero to a prototype I can show is now very fast. Absolutely. And if you put some restrictions in place, some things are then, if I was hearing you correctly, you can then actually take out to production. Some things actually do, are able to ship relatively easily, but others are not. So I guess the place I would start to ask you is like, how do you draw those lines?

33:34And are there ways in which having, for example, a database that can handle all these different pieces together makes it easier to bridge that gap? Yeah. And I would obviously be biased because of working at MongoDB right now database. And we are not vibe coding a database. I mean, there's too much at stake. You're not? No, we are not. We are not vibe coding. No, that doesn't mean that we are not leveraging AI for many things. for prototypes, 100%, for this product and engineering alignment. By the way, I didn't answer your question about team organization, but maybe later can put a pin on that.

34:07But also how you handle tickets. Are you on board? I think people, even very senior engineers, when you ask them, say, oh, I have to discover this new part of my code base that I haven't touched in a while. I can go so much faster on that to understand. So I think there are many, many benefits as well for real product. But the code that is written for production, I think that we are still doing it very manually for the core database, for sure. And then I would say even for more internal things or stuff that are maybe a bit less sensitive where you can, for sure, go faster in your... I don't know if Vibe coding is the right thing, but like AI-assisted coding, I still believe like, yeah, the security audit, these observabilities, there's still a lot.

34:48It's not ready yet for this IoT, in my opinion, but they can help you, yeah, prototype and align on tickets and align on requirements. I think that's pretty impressive. So do you have a line in your product of like, within this must be handwritten outside of this AI assist okay? Yeah, all of the core database and core product of MongoDB, we don't vibrate it, we don't AI it. The code is written manually each time by a developer and we are doing the code review as we did and we are doing the security review as we did. That is for sure, yeah. We can use again some AI tools for some security finding kind of things.

35:23I mean, we can get some help, as many companies do. But I would say most of what we experience with AI are a lot of more management tools around that. There's still a lot for, you know, but like all the management tools around that, all the internal tools, all the before coding and after coding piece to go faster. We use AI more and more, but we don't VibeCode as a core MongoDB database for sure. So that's obviously probably pretty different than a very new, not so security-focused startup. So like thinking about that, has it changed anything about how you're internally organizing your teams? Or does it still take just as many people because the core has to be so kind of solid and locked down?

36:06Yeah, yeah, no, that's a great one. I would say so. There's something first as maybe as a cultural kind of stuff that I came to really enjoy about MongoDB. And I saw that in my previous company at AWS as well. Even people that are not engineers are pretty deep, technically. So we have product managers trying to build their MCP experimentation. I mean, I think that's culturally speaking, that's the case. And you can see that because you have people moving from engineering management to product management role easily. And I think I do. So I just want to see that in this context, because I think that if you are in a different context, which can have also pros and cons, but I think maybe that's different.

36:45But I think in that case, it can really fast track the alignment on what you want to build. Because the example you took before, like instead of just describing the long document, oh, that's the UI I want, you can just do it and then you can align and then you can discuss about how to do that. So yes, it does fast track the alignment between product and engineering, no doubt. We went pretty fast in terms of organization. We made the decision a few months ago now to we don't even have an engineering and a product organization anymore. We have a product and technology organization. That's part of it.

37:16It's not just because of AI, but that was part of it. I think two big objectives. One, how to make sure we can fast track the decision making, the alignment, the sharing of information between the product decision and the engineering decision because they are the same eventually. So that's number one. And number two, how to make sure that everyone is customer obsessed. And even if you're an engineer, you should be customer obsessed. And I do believe AI helps with that, by the way. You can really see what your product manager has in mind. We are using that to really show some reports from customer discussion, advisory board, these kind of things.

37:48So we really made this leap because you are touching on the organization. And I do believe that I guess we probably would have done that as well. But with these AI tools and ways to collaborate, I think it's helping product engineering to be in the same kind of smaller teams than before it was a product and engineering organizations. Yeah. No, I think there is definitely something and have been in a lot of conversations kind of talking about this convergence between engineering and product. Whether that looks at more technically minded product people, whether that looks at more product-minded technology people, as code becomes, maybe not in a database core, but in many contexts is more commodity, it means that product mindset is more and more important.

38:27I think that's spot on. I would have a bit of a nuanced take on this one, which is like, I love that product people can be a bit more engineering or engineering people a bit more product or the wording you used. I love that because again, I think that's a great way to be customer obsessed and to be focused on the outcome and what you are trying to achieve more than just trying to articulate what you even think kind of thing. However, the expertise doesn't go away. Like if you're a product person and you are meeting many, many customers a week, you will have an expertise about reading between the lines and understanding the reading the room in a meeting.

39:03If you're an engineering person, it's not just about this prototype that your product manager colleague can build. It's really about how will that scale? How will that evolve? What do we expect my growth to be and my new changes to be and my next security audit to require. So I think, yes, it's great to bring people a bit closer to the other side somehow, under brackets. But I do believe we should respect expertise. There's still a lot of expertise in what it is to build a system at scale for real production usage and what it is to really understand customer intimacy and what is the need of a market.

39:35And we should respect that. We shouldn't believe that because we have tools that are helping a little bit, this expertise doesn't matter. Yes. I think that that is, you know, if I were to summarize one of my big lessons about LLMs is they're incredible tools, but you cannot turn your brain off. Your brain as an expert is super necessary still. Oh, I love that. And to me, I don't remember who said that. I would love to have been smart enough to say it. But someone says something like, oh, I don't understand. My LLM is super smart on all the topics I don't know. But when it's a topic I really know, it's not that smart.

40:11I love that. Because your expertise is still important. When you know deeply a topic, you do realize that the LLM is wrong sometimes. And you do realize that grounded that with, back to the MongoDB case, but grounded that with real data, real knowledge is important. But even overall, I mean, expertise matters. Even within using it, I find I'm better able to guide these tools in areas I know well than in areas I don't. Yeah, I love that. So coming back a little bit to this piece around the data layer under LLM applications, what do you see as the big unsolved problems? What are the things that your team is working on looking forward for the next, I don't know how long we're allowed to project in the AI era?

40:58Two weeks? Six months? Something like that, right? Yeah, it may be. No, I think, you know, everyone out there is developing an AI application, right? I mean, it's pretty rare to see a company that... But you don't have so many... I think we are at this turning point right now where they are really going to production. I mean, you have this famous MIT paper like three or four months ago saying 95 % of these AI applications, they don't make it to production. Or when they make it to production, they disappoint. They don't bring the ROI they were expecting to bring. But I do believe it's changing.

41:32I do believe more and more of these AI applications are getting production ready. and to me the key to your question about what we see, I believe that people, customers really understand more and more how the quality of the grounded response is, how the quality of the information retrieval to make sure that your LLM will not just be LLM smart, it will be company smart, it will know what the company knows. I think the accuracy for use case just being 2 % or 3 % more accurate, that means that you are totally reducing hallucinations. So I think people are really realizing that even a little bit of impact on the hallucination is like a big deal for the user experience.

42:10And I do believe as well that people realize that, well, actually these stuff are expensive. This AI model can be pretty expensive. And even right now, where they are subsidized by a lot of VC money and all of that, that's still expensive. So if you are able to only call the LLM when you need to call the LLM, and when you're able to optimize the length of your prompt because you did a good job at finding the relevant information in your corpus of data before. If you are using embedding and ranking models that are pretty short and cost efficient, that can totally change the ROI of an AI application.

42:44So accuracy and cost of this AI application, I think that would be a big topic for the years to come, in my opinion. In terms of accuracy then, I mean, I think you've talked some about, we've talked about what it takes in terms of searching, in terms of re-ranking and what you surface. Are there any other kind of best practices you've seen or you recommend to folks in terms of what you're actually putting in that? Are you doing pre-processing? How are you navigating those different levers? Yeah, absolutely. I mean, that could be a very long discussion, but I would say, one, the quality of your data.

43:24if you don't clean your data regularly, handle metadata, know what's important. The quality of your data is key. How you are preparing your data for AI? We spoke about shanking. How do you cut your long text into smaller text that makes sense? I think that there is a science, there is a science to an art to that. How do you prepare your data for? So it has to be clean, it has to be prepared for AI. And then that's really the, what is the right information retrieval strategy for you? Do you want results that will be super fast? Do you want results that will be super accurate because you are a legal company?

43:58And when you are providing advice or financial companies that can impact the tax return or legal document, and then you will want to use the best embedding models with all the context and all the lengths, et cetera, and that's okay if they are more expensive. Or maybe if you're an e-commerce company, you will want to go super fast and make sure that you have 10 or 12 good results and it doesn't have to be the only one. So I think it's about your strategy, about what is the right trade-off for you between this usual quality and cost and speed and all of that. Yeah, the interactivity is an interesting one.

44:29Even within an application, you might have different workflows. Like I was talking to someone who was doing an agent and they were saying, yeah, when I know that the user is right there, I bias towards interactivity and speed and getting it up in front of them. And if it's an async workflow, now I care more about accuracy. Now I care more about, you know, I can take my time. Oh, absolutely. And we see that as an example, not a public reference, but we have an e-commerce customer of us and how you make your trade-off for your e-commerce piece, like where the user will go to find their red shoes or Pogbundi sneakers and how you will manage your stock behind the scene.

45:06These are different kinds of requirements and latencies and costs and things like that. So absolutely, depending on the workload, you will have a different sensitivity. So I think a lot of our listeners are now familiar with in some of the AI coding tools, like you can dial up your budget for, oh, I want you to think longer. I want you to reason more. I want the, you know, mini versus high versus whatever. What are the equivalent knobs that you have in the embedding models and the Reranker and all these different parts that you have with Voyage? Oh, that's a great, great one. So I didn't think about this parallel before, by the way.

45:38So I love it. The thinking and the one with the LLM. And you can definitely go fast and cheaper with like, I would say basic, but even the basic, you can have really good ones versus not so good one, but like a basic text embedding model. And we, I mean, one of the value of Voyage model is they come with different sizes. So you can decide how long your embeddings will be. And even the type, like do you want to go with a float or like to a binary kind of, so you can decide how much. So there is definitely a first decision there about a cost versus accuracy trade-off. Then you can discuss whether, you know, we discuss multimodal and context.

46:16These are heavier models. They can take a bit more time, a bit more compute, but they will give you better results. So is your use case worth this investment? And last, we didn't speak much about re-ranking, but re-ranking is an additional layer that more and more customers are using, which is like the embedding model will basically give you, oh, these are like the 10, 20, 100 best documents in your corpus of data for this query. That's pretty fast and you optimize for that. If you want to know the best model, sorry, the best document for this query, then it has to be compute intensive. You have to go beyond the embedding.

46:53You have to go back to the document itself. And that's what re-rankers are doing. So that's also another layer of your thinking versus the LLM analogy, which I like. You can also decide whether you need a re-ranker to have a very optimized ranking of your results. Now, you mentioned for a lot of these in MongoDB, you would put them as an aggregation pipeline. I'm thinking about use cases that I've had in building these things. Oftentimes, I'll kind of do things in layers where I'll show them something quick, fast, but then I might redo behind the scenes. Okay, I'm going to re-rank. I'm going to resurface this.

47:27I'm going to bump this. I'll do things like that. If I were to do that in your system, Can I get those kind of intermediate results streamed out to me in some way? Or like, how does that end up working? Yeah, you can have, well, what you described is like in a single query, you could first go with a quick search and a bit of a longer one. I don't have many use cases in mind doing that. What I have, though, is like a developer that will start maybe the first iteration, they will go with an embedding model and that's about it. And then when you really want to go to production and you will have real users and real data, et cetera, then they will upgrade their model to a more powerful one to improve the accuracy of the result, or they will add a re-ranker.

48:09But thinking out loud, what you described is totally possible. You could totally do a first search and then do a re-ranking in parallel. Like for instance, as an example, I haven't seen it, but it's such a fresh space that I kind of like the idea still. You could still show, oh, these are the 10 pair of shoes that are looking like your red shoes, Nike kind of thing. And you show them immediately. But after a few seconds, you can re-rank them to make sure that the one that is very accurate comes to the top. You could totally build something. The technology doesn't prevent you to build something that dynamically.

48:40I don't know if the financial aspect or balance is there to do that. Yeah, it'll depend for sure on the application involved. But yeah, I think these types of latency trade-offs are all over the place in these types of applications. And so then there is this question of, okay, how much value is there in showing something to the user versus getting the right answer in front of them? And maybe there's value in each. Yeah, absolutely. Absolutely. But at least what's important, I think, is to have options. Whether you use that in the same flow as you were describing, or you use that because you have different phases of your project.

49:14And at one point, you just want to improve accuracy and et cetera. Or you do that because for a specific application or workload, accuracy is so important. And for another kind of workload, maybe you have a free tier and you're okay that your customers have good results. But when they're paying and you have a premium tier, you want your customers to have excellent results. So I think having the freedom of doing that easily because you can change your schema, you can change your AI model, you can optimize these things is what customers are looking for, right? Yeah. And having that flexibility within the same API, same interaction, I don't have to, now I have to go and get a different thing.

49:50No, that's definitely. And same document model. I'm coming back to that. We didn't invent anything new to store the embeddings, for instance. They are part of your document model. They are there. Like that works. And we didn't reinvent the wheel about, because what people love about MongoDB usually is a horizontal scaling and the fact that you can have these shards and replication. Oh, we did the same for search and vector search. You can have your own search nodes and you can decide that they will be a bit more memory intensive and they will not impact your database. So just using the same principles of the document model, of the distributed architecture, just apply to these embeddings, as you mentioned.

50:24Awesome. Well, we're coming close to the end of our time. Is there anything we haven't talked about that you think would be important to discuss before we wrap? No, I think we touched on it, but I just want maybe to double down on it. Things are changing so fast. The quantity of data is changing so fast. It's not just even humans now generating data and consuming data. It's agents generating data. So the quantity of data is changing so fast. The ecosystem is changing so fast. There are new players and some of them are amazing. And some of them are, looks amazing, but won't be here in six months.

50:57It will go super fast. The LLM race is like, I love it, by the way. I love when like, oh, Google is coming with this great one and then OpenAI, but like it's changing so fast. Your sensitivity to, oh, that should be in this cloud provider and that should be on on-prem is changing so fast, I think it's super important to go with a data platform that can handle this flexibility. And that, you know, if something changes, you can change. You don't have to, oh, I have to rebuild this data pipeline and I have to change my schema and I have to integrate this new identity system to these new players. I think it's really important to at least over-index on flexibility.

51:32I love it. Let's wrap there. Awesome. Thank you, Kevin.

51:43Thank you.