Vector databases (beyond the hype)

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:42Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder of Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well today, Daniel. How are you? I am doing so good because a lot of my dreams are coming true in terms of topics to talk about. I've been wanting to talk about vector databases on the show for quite some time. I know that we've mentioned them, but we haven't had a full episode on them. And I was scrolling through LinkedIn and saw a set of amazing posts and very practical posts about vector databases that I quickly shared and also sent a message to Prashanth Rao, who is a senior AI and data engineer at the Royal Bank of Canada.

1:36Welcome, Prashanth. Hi, thanks a lot for having me. Yeah, yeah. Well, now you have a three-part series on vector databases, a three-part blog series. What makes each one different? Understanding their internals and not all indices are created equal. I hope we can get into a bunch of that, but maybe to start out, could you just let us know what a vector database is? And in particular, why are people talking about them now? For sure. So I think the way I want to answer this question is I'd like to break it down into parts and answer each bit sequentially. Great. So to answer what a vector database is, let's start with what data is in the first place, right?

2:19The definition in my head is data is an organized collection of structured or semi-structured information, and it's stored digitally in a computer. Now, when you have data, you need somewhere to put it. So that brings us to the question, what is a database? So a database is a system that's built for easy access, management, and updating, and also querying the data at hand. We also need to talk about what vectors are. Vectors, you could call them a sort of compressed data representation that contains semantic information about any underlying entity. It could be text, images, audio, anything like that.

2:54So now we put all of these things together. What is a vector database? A vector database is a purpose-built database that efficiently manages, stores, and updates vectors at scale. I think the scalability is a very key factor there. And it also retrieves the most similar vectors to a given query in a way that considers the semantics of the query. So I think all of these terms holistically come together to form what we understand as a vector database. And when you say semantics, what do you mean in terms of semantics and how that maps onto a vector? So I'm sure everyone, most listeners are familiar with the concept of language models.

3:34NNNs are everywhere. So the thing with semantics is typically a query that you have. Like if you write a query to like a search bar on Google or something, you're thinking in terms of keywords. You're just thinking in terms of, okay, I want this particular thing, this item, whatever you're thinking about. and you type the word in there. Where semantics comes in is, did you type in something along the lines of what the data itself has? So can the query actually translate into something that the database actually understands and produces the result that is most meaningful to the query that you put in?

4:10So it's not just about the words or the features of that word, it's also about the meaning of that word and how that comes together in the underlying internal of the data. Cool. Well, so before we keep going, just because, you know, you have developers and data scientists, and they've worked with kind of all the other database types that most of us have worked with for decades. And we have multiple times over the years had to kind of like understand the new thing that's out and what the, you know, what the value is. No SQL. There you go. And so like, I'm going to jump on that. You know, we started with the SQL query language with these that are for relational databases.

4:47and then we went to NoSQL, which there are variants of and things called object databases. I understood your definition of vector, but I didn't understand how it related to the utility or lack thereof in some of those other approaches. Could you kind of lay the groundwork or the landscape of what that is? For sure. So yeah, I mean, I'm a total database junkie. I love thinking about the various kinds of databases out there. So actually, before we go into that, a quick summary in terms of where I'm coming from, right? So I started off as a data scientist. So I'm fully in your world, Daniel. And it's been a few years down that road for me.

5:25And I think for me, I've hit that point where I've been lost in the world of models and hyperparameter tuning and absolute data scientists will relate with that. But the more I began thinking about it, there are people who have entire PhDs in database theory, right? And their implementation. But then the more I've worked with data, I realized that you don't need a PhD to understand enough to build a working application built on top of a database. That's when you began thinking about what exactly are these different flavors that you have out there. I mean, of course, we've all come across SQL databases at some point in our careers if we worked in tech.

5:59So to answer that question, I think the general history of how these things panned out is quite interesting. I believe the origins of SQL databases come from way back in the 70s, I think, when this field called relational algebra was formalized. it's a kind of like you could say a formalization of the mathematics around what it means to join data query data store data in a database in a way that is queryable i think sql databases are so mature so tried and tested and the reason they've withstood the test of time is because they view the underlying storage or the underlying data as structured and in many cases you have structured data that is in the form of transactions and what a transaction basically means is some event happens in the real world and you log that information and you essentially build up a sequential chain of data which is basically a table and that's kind of what the relational data model came from and where relational models get interesting is you have tables that are related to other tables and that kind of maps into real world complexities where not all data is independent like some of the data depends on other things like a person's metadata could depend on what company they work at and things like that.

7:10So that's how relational data kind of became the norm. People were gathering data from digital systems and then putting them together. And SQL became the sort of standardized query language that you could use to query data. Fast forward to mid 2000s and the NoSQL movement starts to pick up. And where that comes from is there's a point beyond which relational data modeling can become a bit inflexible. Like it becomes a bit rigid because in the real world, you have data that comes in from various sources. Now, some of that data can come in very rapidly. With the advent of big data and streaming and all these rapid ways of gathering data that we have today, it became very obvious that the schema-based approach, a schema is basically what kind of data types exist in your table, right?

7:55So the way relational models were built was you needed to define a schema, and the schema kind of was the ground truth. The data has this data of this type only and that's what you expect in there all the time i think the no sql movement's sort of built on top of the limitations of the relational approach of being pre-decided by a schema because to be truly flexible in terms of the massive amounts of data coming in from various systems you need to have a schema-less approach at times and the schema-less approach basically means you store documents you dump data in semi-structured json blobs and things like that in a scalable way.

8:31And I think horizontally scalable became very, very important in that period. The earlier databases that were relational, I think they were more vertically scalable in the sense that you could just add more and more compute and you essentially scaled up your data that way. But now with NoSQL, the idea of distributing the data as documents across multiple machines and having those machines communicate with one another, that became a new paradigm. But I think the challenge with NoSQL is because of the underlying nature that the data need not necessarily be dependent on itself, like in the sense of relational tables, they didn't adhere to the SQL language standard.

9:08And they kind of diverged. MongoDB was among the first, and there were many others that came after it using JSON-based query languages. So there was a big bifurcation, I guess, in, you could say, the database community when on one hand you have SQL enthusiasts who swear by the declarative nature of SQL, and then you have the other community, NoSQL, who uses JSON, essentially, to query the database. They claim it was developer-friendly, and JSON is a developer-friendly interface, language agnostic, and so on. So in some ways, it does have its benefits, but then depending on your use case and depending on what you're trying to do, there are people who will argue on both parades that SQL should be the only thing you should use or no SQL should be the only thing you should use, and so on.

9:47So does that clarify aspects of both those camps before we move into the modern ones? It does, and then if you could distinguish as you go kind of how Vector is different from those others that would be helpful for me, I know, and maybe some other folks in the audience. Yeah. And I think maybe one thing that I loved about your blog posts is I see some of the players from the world that we just talked about represented within that landscape. And then also some that I'm not familiar with, or at least that I've seen only recently. And so you've got these different axes like Postgres, which is a SQL based query language to a relational database has some part to play in this vector database ecosystem.

10:34But then others which seem to have their own query language. So maybe you could also start to break down for us. So we want to store vectors in databases now to do these sort of semantic queries. Does that need to be soared in one or the other of these types of databases that you've talked about developed over time? Or how has that happened? And what are the sort of major categories of players in the vector database space? Absolutely. So I think, yeah, before we get into the specifics of databases, I think to answer Chris's point, we definitely do need to talk about the evolution, right? I see that vector databases are a natural evolution of the NoSQL class of databases.

11:14If you imagine a Venn diagram, you have like a circle that represents SQL and the other circle represents NoSQL. You have an intersection, that intersection point, I believe they're called NewSQL now. I'm not sure if you've come across that term. It's quite interesting, but NewSQL, they technically use SQL-like languages, but they also claim horizontal scalability and a bunch of other things related to asset compliance and all the other things. So it marries the benefits of both SQL and NoSQL paradigms. I was thinking initially, where do I place vector databases? does it go in that intersection or does it sit purely in the no sql camp then i imagine this as you extend that circle that has no sql it becomes like a blob like a fuzzy amorphous blob no sql is huge and i think in my head vector databases are like an extension to no sql and why they came about uh to understand what vectors are how they're stored in database i think it's important to understand what search is and what essentially you're doing when you query a no sql database So where it comes from is in the early days, I guess people were just submitting an exact query using a JSON sort of query language like how MongoDB has.

12:22And that query has to have all the terms or parameters in there that tell you what you want to fetch from the database, right? In a SQL world, it will be done with a declarative query in SQL, whereas in NoSQL, you typically do it in JSON. Over time, I think the idea of full text search became very important because I think everyone wants to be able to retrieve information from massive blobs of data fitting around. And how do you query that, right? If it's in a NoSQL sort of format, if you can't write a SQL query to retrieve it, how do you get that information? So the idea of a full text index came about.

12:55And what essentially that is, is it uses a concept of inverted indexes, inverted file indexes, sorry, where you consider the term frequencies of terms that appear in a certain document and obviously the relative frequency of how often those terms exist in a document versus the entire data set. So you combine all those things together, similar to how DFIDF is in data science. There's an algorithm called BN25, which is the most popular inverted file index algorithm. It's the most commonly used one for full text search. So the early days of search involved how do you scale that up because you have massive amounts of data, how do you build that index very, very efficiently?

13:32And then the querying interface sits on top of that so you essentially submit a query saying okay i want so and so term in the keyword that you put in and the inverted file index the bm25 algorithm it considers the words frequency and it considers subword features and a bunch of other things to intelligently retrieve relevant documents that contain that term while also throwing out you know useless words stop words and things like that so it was more of like a bag of words sort of which we consider an NLP analogy, it's kind of like a bag of words way of approaching text. Now, fast forward a few years, I think ever since the transformer revolution happened, people began observing the obvious power of transformers in encoding semantics, right?

14:15A transformer is way better at isolating meaningful terms in a document, especially when you're doing things like classification, retrieval, and so on. So how could you merge those benefits of a transformer with what you have in a database so i think vector databases the term got coined i think much later after transformers came about it was mostly called search engines before that a more generic term i think a catch-all term for anything that in word search but nowadays i believe search engine refers to a more like you consider semantics as a key component so essentially vectors are the only thing that can do that so to really describe what a vector is essentially you have a language model typically a transformer based language model that you use to embed the representation of a sentence into tokens and the representation is stored as a vector the vector that you have essentially for a particular sentence typically those are done using sentence transformers which is the most common kind of model you use that essentially embeds the entire semantics of that sentence in the vector and then the way this scales up is you consider the context of each and every token in that vector in a way that when you submit a query, the semantics of the query are mapped to the vector in your database, and you can find a similarity between what you entered as a query and what exists in the data.

15:36So a vector is a very powerful way of, you could say, compressing the representation of meaning in a sentence or a document in a way that scales up numerically, and you can rapidly query that in a digital.

15:56This is a Changelog Newsbreak. We've talked about prompt injection quite a bit since ChatGPT ushered in the LLM era. In brief, that's where you handcraft a prompt that tricks a chatbot into not following its own rules. Well, new research has uncovered some new LLM attacks on the block which aren't exactly that. Quote, large language models like ChatGPT, BARD, or CLOD undergo extensive fine-tuning to not produce harmful content in their responses to users' questions. Although several studies have demonstrated so-called jailbreaks, which are special queries that can still induce unintended responses, these require a substantial amount of manual effort to design and can often easily be patched by LLM providers.

16:45This work studies the safety of such models in a more systematic fashion. We demonstrate that it is in fact possible to automatically construct adversarial attacks on LLMs, specifically chosen sequences of characters that, when appended to a user query, will cause the system to obey user commands even if it produces harmful content. End quote. The biggest difference here is that they're achieving the jailbreak in an entirely automated fashion, and they make a case for the possibility that such behavior may never be fully patchable by LLM providers. Game over, man. It's game over. What are we going to do now?

17:24What are we going to do? You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news. worth your attention. Once again, that's changelog.com slash news.

17:52So Prashant, you kind of alluded to this, and I think that explanation was amazing of how this vector-based semantic search really exploded around the time that transformers and large language models did. I think even in this past, let's say, year, there's been this huge explosion of interest in vector databases. Could you maybe describe a little bit, so we know that you can search a vector database to find similar statements, let's say, or similar chunks of text where the similarity is based on semantics. How are people using them with regards to their AI workflows? And how does that kind of correspond to what's sort of popular right now in terms of what people are exploring with AI?

18:40So yeah, I think I need to highlight the fact that I'm both fascinated and frustrated by the current state of marketing in vector databases, both at the same time. I'm genuinely interested in the use cases, don't get me wrong. When combined with LLNs, large language models, like ChatGPT, you could say any sort of language model layered on top of a vector database can be used to build some very, very interesting applications. One of those interesting applications is querying your data via natural language. I think this has always been a dream of data scientists and people who work with data, right?

19:12Rather than writing my query by hand or constructing the query painstakingly from the ground up, can I just talk in natural language and have the database kind of respond to that query in natural language as well? The application we built using an NLM at the core, and essentially that would be powering the whole translation of human instruction to machine instruction and back to human. I could go into the details of specific applications, but one thing I do want to really throw back at you is, I know this is a practical AI podcast, right? So I guess what I was hoping to get into is I have an idea for a fourth blog post and in the series, basically.

19:50Part of it is the trade-offs, right? What really interests me about the various vector databases out there and why I began writing about these at length is when it comes to understanding what tool to use in the real world. When you have a business problem, when you have a particular case you're trying to address, obviously there's tons of information out there. You could go out and read a bunch of blogs and papers and come up with your trade-offs. But I think it makes sense to actually walk through some of these trade-offs. And my understanding is that as you go through these trade-offs, you actually begin formulating the value of these things much more clearly.

20:25And in my head, I think it makes sense to talk about the use cases once we go through some of these key trade-offs because in many ways using a tool depends on what goes into it and what you thought about the different options. You can dive right in because yeah I had follow-ups which were essentially what I think you're about to cover anyway so I'll just leave the mic with you man. For sure yeah so basically it makes a lot of sense to write about this and obviously read it at your own time but this is a great place for me to begin talking about it and eventually I'll put these down in words as well.

20:55So I've broken these down into I think roughly eight categories and the trade-offs. I'm specifically speaking about what do you need to think about when you're thinking about a database. And this will answer exactly what you talked about earlier, Daniel. So the first thing I think Daniel mentioned is the idea of deciding between existing databases that have been around, document format and things like that, versus newly designed databases specifically for vectors. So I'm going to call it purpose-built vendors versus incumbent or existing vendors. right so i think it's very important to understand in many cases you might just be looking to add semantic search capability or just retrieving information using semantics on top of an existing application right and that existing application could very well be built on a well-known tried and tested solution like elastic search postgres and so on there's many solutions out there and obviously in those cases it makes sense to just say hey but why can't i just leverage the vector index or the vector storage of that database itself like for example you mentioned postgres one real big concern with this is if you look at some of the material online on the performance of these the methods of pg vector pg vector is basically the vector plugin add-on to postgres and there's been enough documentation about this but it essentially the way it's been slapped on to postgres is as like an add-on it's like built by a third party called superbase and they add a vector functionality to the existing engine that Postgres has.

22:25So by its very nature, because it's not tightly integrated with the underlying internals of the database itself, like the storage layer, the indexing and all of that, you're going to miss out on a lot of optimization. Not you, but the technology is basically not optimized from the ground up to speed of indexing, performance during querying and so on. And this has been well documented. So that is a very big concern. depending on your use case and how much accuracy and what quality of results you want are you better off using an existing database that you already have in your stack or actually bringing on a new tried and tested purpose-built database for that very reason right and from my experience i've been tinkering around with quite a few options out there with purpose-built vendors in my opinion they are always a better solution in terms of scalability efficiency and also accessing the latest technology, like the latest algorithms out there, what indexing algorithms are out there, how do they get the best bang for buck in terms of your speed of indexing, the quality of query results, the latency of those results, and so on.

23:26So I feel like in the long term, if you actually are serious about building a vector search or large-scale information retrieval system that considers semantics, it makes far more sense to think about a purpose-built solution. Many, many database solutions are out there. I've listed some of those on my blog. And I think those are going to win out over the incumbent vendors who have kind of built, you know, vector offerings, as you can call them. What we're talking about is exactly what I had hoped we would talk about in this episode because your blog posts were so practical. In terms of how you think about the infrastructure that you work with day to day, would you recommend, because sometimes you don't know how much you need to optimize at the beginning and you can over optimize, right?

24:11So would it be a valid maybe stepping stone to say if I'm already working with Postgres, I could try out the vector capability of that. And if it works for my use case and I don't have, you know, three million documents that I'm searching over, maybe it's maybe it's fine. I just have a you know, I'm doing my personal blog or something and then kind of optimize as you hit a wall. Or is there danger in kind of trying to make that, put a square peg in a round hole sort of thing and get yourself in trouble? You hit the nail on the head. I was going to exactly say put a square peg in a round hole because I face those issues myself.

24:51I wouldn't name exact database vendors, but I work with SQL and NoSQL databases, which obviously have vector solutions. I think the challenge and the issue with saying that, okay, I already have something that works is you've got to remember that every single database that has existed for i think more than 10 years the databases come with baggage and they have their own tech debt that is associated with the underlying programming language they're built on there's years of decision making and architectural decisions under the hood that they've taken to implement solutions the way they have so they can't just throw all of that away and then build a vector solution that is optimized from day one right it's going to take a fair amount of time before these incumbent vendors are able to optimize their offerings to a point that perform as well as purpose-built vendors because these purpose-built vendors have spent thousands of manors, I guess, per offering in just tuning and building for a very specific goal.

25:44So what I've noticed in my experiments is that a lot of features that you take for granted in a purpose-built offering are not even available in the existing solution. Like PG Vector is a very, very young solution right now. Elasticsearch is vector offering. I've worked with that as well. it is also i mean considering elastichurch has been around for so long they only released their first vector like ann algorithm i think last year like 2022 so in terms of a database's uh capabilities that's very very young so i would say like there's a lot of things that you could potentially be missing or lacking and i'll cover some of those in my other trade-offs that i missed as we go forward yeah yeah let's go on to those i'm curious what number two is for sure the number two is uh i came across this in my first blog and reading some of the comments on them and one of them brought up this fact that the trade-off between using a database that allows you to build your own embedding pipeline or versus using a built-in hosted sort of embedding pipeline and by that i mean how do you generate these embeddings or these vectors right many people are familiar with sentence transformers it's available on hugging face and a bunch of other open source platforms so essentially it's quite easy or you could say it's trivial to put your data into these pipelines and generate sentence embeddings that you can just use to ingest into a database alongside your actual data.

27:03So you have your document data that has all the fields and attributes that you have in there alongside the vectors that encode the useful information in that that you want to query on. So that's a relatively trivial thing to do. But there are certain database vendors who offer convenience features on top of that where they embed the API of these models inside their own offering. So if you're just getting started and you don't know much about how vectors work or how, you know, NLMs work or any of these things, that might be something to consider. You might be better off using something like VV8, which has pipelines built in where you can just tell it, okay, connect to Hugging Face so-and-so model, and it will build the embeddings for you, as opposed to you writing your own custom transformer pipeline that actually takes in the vectors, generates the vectors, and so on.

27:48Now, if you have experience with transformer models, you might be far better off in doing all of the embedding work upstream, paralyzing, you know, and optimizing that portion, generating those at scale, and optimizing from a cost perspective, getting those done with the least resources and, you know, most quality that you can, and then just sending the vectors over to your database. So this is an important thing to consider depending on the level of experience that you, your developers have in your team, to actually bring the vectors in. Gotcha. That makes perfect sense. What are some of the other trade-offs?

28:22So then the other thing is the two key stages, right? You could say you could break down when you use a vector database as a developer. The first stage is the input, which is essentially building the index, right? I go into the indexing methods in a bit more detail. That's not really a trade-off. It's more about knowing what indexing even does under the hood. But what indexing means is you have data that you need to encode into a vector, right? Now, it's not as simple as just dumping a vector, which is like an array of numbers onto your database. You have to be able to search through those vectors.

28:56So the goal of indexing is to design efficient data structures and store the vectors using those efficient index data structures in a way that they can be queried efficiently and at scale. So that is an upstream process. You do that once upfront, you bring all your data in, it's indexed, and now you have a bunch of vectors in there that are searchable. the downstream portion of that is querying right it's basically like inference in nlp the query stage involves you taking the user input transforming that into a vector just like you did your raw data and the vector embedding that you use there is an embedding model that you use to basically transform your data so that they are compatible so that's a downstream step you're clearly separating the indexing step from the query step right so the trade-off here is is your database optimizing for indexing speed or query speed or is it mature enough that it has optimized for both and if you look through all the offerings out there many of the existing vendors have focused more on one end of the pipeline and not so much on the other some of them are faster at indexing and not so much at querying but some of them are way better at querying and much much slower during indexing so generating that index actually can be a very expensive step because it's not only about using a sentence embedding model or a transformer it's also about the database being able to translate those vectors into an index and it can actually query so depending on the size of your data this could take hours or even days like it's not unheard of here of indexing the uh periods of the order of days um and of course depending on your the amount of money you're throwing at it you could go use gpus to speed up the vectorization and and use multiple you know parallel instances of the database to scale that portion up.