Improving search with RAG architecture with Pinecone CEO Edo Liberty

0:05Hi, listeners, and welcome to another episode of No Priors. Today, Alad and I are talking with Ido Liberty, the founder and CEO of Pinecone, a vector database company designed to power AI applications by providing long-term memory. Before Pinecone, Ido was the director of research at AWS AI Labs and also previously at Yahoo. We're excited to talk about the increasingly popular RAG architecture and how to make LLMs more reliable. Welcome, Ido. Hi. Okay, let's start with some basic background. Can you tell us more about Pinecone for listeners who haven't heard of it? Like, what does it do and how does it differ from other databases?

0:40So Pinecone is a vector database. And what vector databases do very differently is that they deal with data that has been analyzed and vectorized. I'll explain in a second what that means by machine learning models, by large language models, by foundational models and so on. The most large language models or foundation models, actually any models, really understand data in a numeric way. Models are a thematical object, right? And when they read a document or a paragraph or an image, they don't save the pixels or the words. They save an numeric representation called an embedding or a vector. And that is the object that is manipulated, stored, retrieved, and searched over and operated on by vector databases very efficiently at large scale.

1:35And that is Pinecone. When we started that category, people called me concerned and said, what is the vector and why are you starting a database? And now I think they know the answer. How did you think about this early on? Because you started the company in 2019. At the time, this wave of generative AI hadn't happened quite yet. And so I was wondering what applications you had in mind, given that there's so much excitement around Pinecone for the AI world. The prior AI world had a slightly different approach to a variety of these things. And I'm just curious, were you thinking of different types of embeddings back then?

2:12Were you thinking about other use cases? What was the original thinking in terms of starting Pinecone? The tsunami wave of AI that we're going through right now didn't hit yet. But in 2019, the earthquake had already happened. Deep learning models and so on have already been grappled with. Large language models and transformer models like BERT and others started being used by the more mainstream engineering cohorts. You could already kind of connect the dots and see that where this is going. In fact, before starting Pinecone, I myself had founder anxiety between are we already too late versus nobody knows what the hell this is and we're way too early.

2:55And it took me several months of like wild swings between those two things until I figured maybe the fact that I have those too early, too late mood swings maybe means it's exactly the right time. Maybe, Ido, you can actually just expand a little bit about, you know, in what use cases people want to use embeddings, right? I think there are ways to interact directly with language models and then reasons for, for example, reliability or context length like that people and performance that people interact with embeddings in like a RAG architecture or in semantic search. So maybe you can sort of talk about some of the driving use cases.

3:35I mean, the obvious way, in some sense, to add knowledge to your conversational agent, whether it's chat or what have you, we talk about it as generative AI now, but it's much more general than that, is to, again, not shockingly bring the relevant information into the context, right? So that you can actually arm the foundational model with the right pieces of content, with text, with images, with what have you, right? You want to be able to retrieve that from a very large corpus of knowledge that you have, whether it's your own company's data or whether it's the internet or what have you. It so happens that LLMs are already very, very good at representing data in the way that they want to consume it, which is these embeddings.

4:26And so you can, at question time, in real time, okay, at the time of the interaction, go and find relevant information. And relevant might be associated with or correlated with or something that is similar to whatever it is that you're being asked about. And once you bring that into the context, you can now give much more accurate answers, right? As a side experiment, we actually loaded what's called Common Crawl, which is the top internet pages crawled fairly frequently. We load that into Pinecone and saw what happens when you augment GPT 3.5 and 4 and Lama and Mixtral and models from Coherentropic.

5:16And you could see that if you augment all of them with RAG, even on the internet, which is data that they were trained on, you can reduce hallucinations significantly, up to 50 % sometimes. Interestingly enough, many of them actually start behaving quite similarly in terms of level of accuracy, even though without RAG, they actually have quite different behaviors. So it's sort of both like a uniform improvement and a little bit of leveling the playing field. Now, you know, because we know we can do that very well now, now you can do that also with proprietary data, with data inside your company and so on, stuff that, of course, is not available on the internet and stuff that those models were never trained on.

5:57And interestingly enough, again, the quality ends up being incredibly high. I assume most Pinecone users are not, you know, using LLMs and retrieving against general internet data. Like, what kinds of companies were your earliest or biggest users? Like, what kind of data do they want to retrieve against? So most companies do use their own company data. It could be whatever it is, depends on the application they're building. It could be legal data, medical records, internal wiki information, sales calls, you name it. There's an infinite variety. I want to say that this is just rag. I mean, this is just semantic search.

6:35I mean, there are many other applications that we didn't talk about, but we can keep it focused on this application for this conversation. And is it dominated by a specific use case? Like, were there customers that you feel like really represent the Pinecone use case well? Yeah, 100%. First, text is probably most of what we see. Nowadays, models are really good at images and so on, but text is still the predominant data type. Notion Q &A now runs on Pinecone, and they serve essentially question answering with AI to tens of thousands and probably hundreds of thousands of their own customers. Gong does the same thing with sales calls, again, serves all of their use cases for all of their customers and so on.

7:21So one of the most common patterns is companies that themselves become trailblazers and innovators with AI, and they themselves hold a lot of their own users' or customers' text, and they want to search over it or generate information on top of it. That ends up being an incredibly common pattern. I guess earlier this month, one of the things that Pinecone announced was the serverless offering called Canopy. Could you tell us a little bit about why you decided to go down the serverless direction and how you view that in terms of either use cases or adoption or other things? So Canopy is actually an open source that we put out there as a framework for people to learn how to use RAG.

8:03Pinecone serverless is just called Pinecone. It's just Pinecone, but serverless. What it does is basically removes the limits from what people used to experience before. When we started Pinecone, a lot of the applications had to do with recommendation engines and anomaly detection and other problems where usually the scale was actually fairly small. And the requirements had to do with super low latencies and sometimes high throughput. and as a result, you still see a lot of databases kind of play in that field. We very quickly figured out with our own customers and our own experimentation that something else is much more significant with just a scale and cost.

8:54If you want to be able to answer correctly, you just have to know a lot. If you want to do that, you have to ingest hundreds of millions, billions, sometimes tens of billions of vectors into your vector database. And you want to query it efficiently in terms of cost. You just don't want that to explode in terms of, again, spend. And finally, you want to do that easily. So you don't want to spend weeks and months setting things up and getting it to work. And doing that in our old architecture and, frankly, with any other architecture today that's not serverless is very difficult. And serverless is here to basically resolve those main problems.

9:38It's incredibly easy to operate. It scales massively. I mean, again, there's no theoretical limit to how much it can scale. We've tested it with tens and tens of billions with live customers and live traffic. And I'm not going to go into the architectural design, but it's actually designed to be incredibly efficient, like asymptotically better than what can be done with any other architecture. It's fundamentally about removing all limits so people can actually have all the information they need ready for the foundational models. You mentioned Canopy is to help enable more people to build RAG products.

10:18Where do you see developers or your customers struggle to get embeddings-based AI products generally successful? Or what were you trying to achieve with Canopy? Yeah. So, the vector databases and Pinecone specifically are very foundational model, are very foundational pieces of technology. We're very deep in the stack. And to build a proper full end-to-end solution, say like Notion Q &A, there's quite a lot that you have to build on top of it. You have to ingest documents and what's called chunk them. You have to figure out how to break them into factoids and pieces of information. You have to embed everything with models.

11:00You have to ingest them into the vector database. When you get a query, you have to figure out how to manipulate it and how to embed that. You have to search over it. You have to re-rank. There's a lot. There's a whole system you have to build around it. And a lot of people told us that this is actually quite complex and they're right. We put out Canopy as really an example. It is an end-to-end kind of cookbook. If you just take this, it should work. You should probably, once it works, you should figure out how to make it better for your own application, right? Because, you know, medical data is not JIRA tickets, you know, and JIRA tickets are not Slack messages and you might be building a different product.

11:42But at least you have some end-to-end starting point that already does something and you can start improving on. Two of, I think, the most common comparison points for vector databases that people use are, A, like traditional databases, right? Like why not just use Postgres, PG vector or some index associated with an existing database? Or B, like sort of more traditional search or incumbent search technologies or services like Elastic or Algilia. Can you talk about like, you know, why not other databases or like how you think about traditional search? Yeah, I'll just go back to the fundamentals about what are you trying to achieve, right?

12:23And what we're trying to achieve is to give as much context and as much knowledge to foundational models as possible. Do that easily at scale, you know, on a budget, get to a unit economics that actually works for your product, which is incredibly hard to deal with AI with like many discussions going on about that now. Those other products don't work. They don't work either because they don't scale in terms of the efficiency, scale, cost, the trade-offs that they can offer. Because they're not designed to do this. They're designed to do something else. They kind of thought about vector index as a bolt-on, you know, retrofitted feature.

13:10And so, yes, it works at small scale. But when you try to actually go to production with it, you understand the limitations. With other search technologies, again, this is the wrong search mode. If you're searching with keywords and just not finding the relevant information, because the embeddings, the contextual space in which these pieces of text, documents, or images live is in vector space, in high-dimensional numeric space, not in keyword space. And like everyone that's ever searched the inbox for an email, you know for a fact you have and not find it knows that keyword search is a deeply flawed retrieval system.

13:54I'm just curious if customers or developers are trying to combine the existing search systems they have. I know you also are increasingly supporting hybrid search. So kind of wanted to understand that. Where are embeddings amazing and useful and delivering new experiences and where they're not enough or not like the full experience that end users want. So it's interesting. Our research actually shows that when you do this well, you very rarely need keywords alongside embeddings. But getting embeddings to perform perfectly is actually, it could be quite intricate. And we find that it's very convenient to have keywords alongside embeddings and to score those things together.

14:42We call this hybrid search. And in fact, we made this even more general and we said, okay, why not? Keywords under the hood are actually represented as sparse vectors. That's true of any keyword search, by the way. This is just kind of the mathematically identical. And then we said, why don't we just make this more general and just say, hey, you can give either sparse or dense vectors or both of them and kind of have the best of both worlds. and people find that very convenient. And so I'd highly encourage people to look at it and improve by boosting and all sorts of other tricks that you can bake into spouse vectors, including keywords.

15:20My guess is that that's not going to be the dominant mode of search in the very near future. You think we progress, like you think hybrid search is a more temporary convenience? I mean, I think it'd be used for boosting and other types of levers to control your search. I think the mode of you baking keywords into that is going away. Yes. And maybe just going back to like the traditional database market, like why not in my Postgres or my long or whatever I'm using already? Again, I mean, we see this in the market. A lot of people tell me, hey, I already use tool X or database Y and why not? And frankly, oftentimes when it's some tiny workload, you're just learning how to use embeddings for the first time and so on, it might actually work okay.

16:12It's when people try to actually do something in production, they're trying to scale up, they're trying to actually push the envelope, or they're trying to launch a product that needs to have some unit economics attached to it that makes sense for that product. That's where people run into huge problems. And so many of them just, you know, start with us to begin with. To be honest, a lot of them are enthusiasts and they actually kind of enjoy learning how to use a new kind of database. And are you, you know, user experience is smooth enough and, you know, so many tutorials and notebooks and examples that they actually find it exciting.

16:48But I guess some don't. And that's fine. So maybe one more on database dynamics. Pinecone is closed source. It's gotten great adoption, but many databases and mature market are open source. How do you think about this decision? And has that been an issue for you? I'll say that most databases started before cloud was really a fully mature product or market or platform. Okay. And so that was the precursor to PLG, essentially, or whatever. It was PLG, right? That was the only way to put a technically complex product at the hands of engineers, was to open source it. And you see, I think, all, I mean, maybe not all, but definitely the larger databases that are open source out there, I think that's the reason they did that.

17:46When we started Pinecone, we asked the very basic question of why do people open source the platform? One of it was to earn trust. One of them was to get contribution from the community. And one of them was a channel to users. And we figured we can earn trust by being excellent at what we do and providing an amazing service. We don't need external contribution. And in fact, if you look at statistics, even companies that are open source, 99 % of the contributions are actually from the company itself. Not 99, but high 90s. And so that doesn't actually make a huge difference. And in terms of experience, we figured that we can actually provide a much better experience and much better access to the platform than what open source does.

18:41And Pinecone is a fully managed and multi-tenant service. And to be able to run that at scale and provide the cost-scale trade-offs, we actually run a very, very complicated system. And in some sense, even if we gave it as open sales to somebody, they wouldn't know what to do with it. It will be a Herculean effort to even run this thing. The right decision was basically that we should offer this as a service. We should manage it end-to-end. And as long as you give people a fully reliable interface and you keep doing that year after year, you earn the trust and the ease of use that open source becomes, I hope, not an issue.

19:19It's funny because the two anecdotes along those lines, I remember talking with, I think it was Ali from Databricks, and he said that if you can avoid doing open source, you should. He felt like it was an incremental challenge because you get distribution through open source, but then you have to figure out the business model. And so he viewed it as like, you know, I think that the analogy he uses is like making an open source project work is like hitting a hole in one in golf. And then you pick up a baseball bat and you have to hit a grand slam because then you have to do the second act to make sure that the thing actually works as a company.

19:52That's right. No, I mean, I agree 100%. I mean, this is exactly what we're experiencing. And in fact, we already see, even though new players in the vector database space that basically started to try to take us down, all took the open source angle. We already see them, even young as they might be, they are already struggling with their open source strategy. Serverless is the fourth, almost complete rewrite of the entire database at PineCon. Yeah. Yeah. The one other thing that's coming in terms of the LLM world, which may or may not impact you, and I'm sort of curious how you think about it, is increasing the long context windows for foundation models.

20:33Does that change how people interact with embeddings in vector databases, or does it not really impact things much? There's things people are talking about in terms of infinite context or other things like that. I mean, I don't know what infinite context means, to be honest. It's like very big. it's infinite it's like huge oh oh thank you yeah yeah all right you're welcome first of all those companies sell uh their services by the token so the fact that they allow you to use infinite context windows is not shocking okay uh that's good for business The second thing is there's plenty of evidence that increasing the context size doesn't actually improve results unless, you know, you do this very carefully.

21:22Right. So just what's called concept stuffing is not helping. You just pay more and don't actually get much for it. And the last thing that even that, even if you kind of buy in to the marketing, that runs its course. Right. If you're, it's like saying, oh, I don't need Google because I can, every time I query Google, I can send the internet along with my query, right? It's like, yeah. Well, theoretically, that's maybe possible, but clearly, practically, that's not feasible, right? So, at some point, the context window just becomes gigabytes and gigabytes and gigabytes of data, like terabytes.

22:04I mean, where do you stop, right? And so already today, we have users who use not even very large models, maybe a few billion parameters. And the vector database next to the model contains trillions of parameters. And they get much better performance that way. Just attaching all the context to everything you do, I think, runs its course very, very quickly. And it's also unnecessary, to be honest. Yeah, I guess related to that, another place where people have been talking about embeddings and vector databases is in sort of aspects of personalization and privacy. And I'm a little bit curious how you think about that, because one of the risk people view as running an LLM over a large data corpus or fine tuning it against a specific company's data is the issue of data leakage.

22:59You know, say for example, you're an HR company and you don't want different people's salaries to leak across an LLM because you're using it as like a chatbot to help you with context regarding your own personal data in an enterprise or things like that. Can you talk a bit more about how embeddings can provide personalization and in some cases, potentially other features that may be attractive to enterprises? Yeah. So that's a very common and reasonable thing to be concerned about. data leakage can happen in two main ways. A, if you use a service for your foundational model that frankly retrains their models with your data or records it, right?

23:42Or saves it in some way that is opaque to you, right? That is a huge problem. And I think a lot of people are struggling with that. The second is, if you're building an application in-house, whatever it might be, and you fine-tune your models on added data, that added data might end up popping where it shouldn't in answers to other people's questions or whatever. What people do with vector databases is actually incredibly simple, right? You don't fine tune your model on your own proprietary data, at which point you know for a fact it doesn't contain any proprietary data because it's never seen any of it.

24:25And then at retrieval time or whenever you apply the chat or the agent, you retrieve the right information from the database, give it as context to the model, but only do inference. You don't actually retrain and you don't save that interaction, at which point that data doesn't exist anywhere. It's like an ephemeral thing. And the added benefit to that is, by the way, that you can be GDPR compliant. You can actually delete data. So if you're a company, like a legal company, and somebody deletes a document, you can just delete it from the vector database. And that information will never be available to your foundational model again.

25:09So you don't even have to devise some complex mechanism for forgetting. You just don't know it anymore. What are the main reasons why people attach vector databases to foundational models? It gives you this operational sanity that is almost completely impossible without it. That's interesting. Yeah, I guess it feels like there's three different approaches that people are using. They're not mutually exclusive for models that kind of overlap in terms of what the hope for output is. One is really changing or engineering prompts or adding more information into the prompt. The second is fine tuning.

25:50And the third would be RAG slash different aspects of embeddings or other approaches like that. How do you think about fine tuning in this context? Like when should you fine tune versus, you know, use some of the approaches that you've talked about earlier? I can answer both as a scientist and as a business owner. As a scientist, I'm all for fine-tuning. We have all the evidence to show that done right, it helps tremendously. As a business owner, I can tell you that it's actually extremely hard to do well. I mean, this is something that unless you have the research team and the AI experts that know how to fine tune, you might actually make things significantly worse.

26:32Okay. So, there's nothing that says that more data is going to make your model do better. In fact, it oftentimes regresses to something significantly worse. With prompt engineering, again, I think it's necessary, especially when you build applications. You want the response to conform to some format or have some property. I think that's a given. You should do that. It runs its course after a while. I mean, in some sense, you get what you get. It's necessary, but there's a limit to what you can do with that. And RAG, I think, is incredibly powerful. But like I said before, when we talked about Canopy, that's not simple either.

27:17I mean, it's simpler than the other ones. but still requires work and understanding, experimentation, and so on. This is almost the hallmark of a nascent market when the simplest solution is still somewhat complex. Yeah, makes sense. What's next for Pinecone? What are some major things coming that you'd like to talk about? So, I mean, there's a ton. We are an infrastructure company, And so we obsess about ease of use and security and stability and cost and scale and performance. Also, as an engineer at heart, I'm very excited about those things. And all of that is coming. Again, serverless is becoming faster, bigger, better, more secure, easier to use.

28:06And we're starting to really grapple with what very large companies and very, you know, kind of trailblazing tech companies are going through. I said that getting AI to be truly knowledgeable is still complex. I think we're starting to grapple with deeper issues that the entire information retrieval community has been dealing with for about 40, 50 years now. We're starting to see those come to the fore in RAG and in AI in general. I guess putting aside Pinecone and sort of the database world and everything else, what are you most excited about in terms of what's coming next in AI? It's hard to say.

28:49I really do want to see a distillation in some sense of foundational models. And by distillation, I know it's a, I don't mean what usually people say, but there's distillation of models. I don't mean that. I mean, the separation of reasoning and knowledge, right? Foundational models get it fundamentally wrong. When we learn how to build the subsystems of AI correctly and for each one of them to do their roles optimally, Either we're going to be able to achieve the same tasks much cheaper, faster, better, or we're going to still want to use the same amount of resources, but achieve much more. What happens today is that we have very crude tools and we try to use everything for everything.