Are Vector DBs the Future Data Platform for AI? with Ed Anuff - #664

0:09All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host Sam Charrington. Today, I'm joined by Ed Enough. Ed is Chief Product Officer at DataStax. Before we get into today's conversation, be sure to hit that subscribe button wherever you're listening to today's show. Ed, welcome to the podcast. Thank you. It's great to be here. I'm looking forward to our conversation. We've got a bunch on the agenda. We'll be talking RAG and vector databases and assistance. But before we do that, I'd love to have you share a little bit about your background. In fact, we've got RPI in common.

0:44Yeah, we do. Yes. Which I think is probably a very chilly place this time of year. So it's been a while since I've been back. Have you been there recently? I wouldn't say recently, probably five years ago was the last time. Yeah, same for me. Great school for those who haven't been there, though. Small, great tech school, but in upstate New York. And one of the reasons why I chose to move out to the West Coast when I graduated was the winters there. absolutely so tell us a little bit about how you got from there to to here yeah so it came out to the west coast really wanted to get into startups and everything that was going on and of course this was the early days of things like internet well it was in pre-internet multimedia and all that but shortly thereafter internet happened and was over at wired in the early days doing the search engine and did a whole bunch of stuff, started a company in the enterprise Java space called Epicentric that had a great run, went on to do some other cool stuff, social media, advertising, blogging, was at Six Apart for a while, the company that made Movable Type and TypePad and ended up part of Apigee, the API management company.

1:58We had a great run there too, did an IPO, got acquired by Google. and after a few years at Google, decided to come over to DataStax, which is the company that makes Cassandra, the Cassandra database, and have been doing that for the last few years. So a bunch of cool, fun stuff, primarily making stuff for people that are building websites, building applications, building content. That tends to be the type of stuff I like to do. I totally forgot about your epicentric connection. I was a very early employee at Plumtree. Yes, that was an exciting time. Awesome. So tell us a little bit about DataStax has been kind of active in helping organizations kind of take on this challenge of using LLMs and RAG.

2:43Tell us about DataStax's kind of angle in that. Yeah, so as I mentioned, DataStax is the company behind Cassandra. And Cassandra was really the original cloud-native database. So an awful lot of companies, whether they're using Uber, whether they're using Netflix, Apple, these are all companies that use the Cassandra database. And when you do something like FedEx package tracking, that's all on topic standards on data stacks as well. And so we knew pretty early on that as people were looking to first with ML and then as AI and Gen AI became a big thing, we knew that that was going to be pretty important.

3:19That people would want to use the data that they had in these systems that power all these interactions. They'd want to add AI to it. And so we looked at how to add the vector capability, the vector search capability to the database. That's something that we did. We did it both within our AstroDB, that's our cloud service, but we've also, everything we do is also in open source. So Cassandra 5.0, it's part of the Apache Foundation, has this vector query capability. And as you know, when you want to get a database to work well with an LLM, and we'll get into it, I think, probably in depth in a little bit when we talk about things like RAG, but the starting point is to allow you to go and have your LLM within some architectures, and we'll talk about RAG, I think, in depth.

4:04But you want to be able to go and retrieve information from the database on a vector-based query. And so what we've done is, and we actually did a couple iterations of this. At first, we implemented, much like most of the other vector databases that you see, we took HNSW, which is the hierarchical, navigable, small worlds approach of adding vector query capabilities. We brought that to Cassandra. We've since switched to something that's called disk ANN, which is a different approach that is more oriented towards optimizing disk I.O. We've talked a little bit about that, too. But the goal here is to go and give you Cassandra, which is one of the most scalable databases for people that don't know, built on technology from Facebook, from Google, from Amazon, and gives you a scale-out data capability.

4:55But how do we bring this vector capability to these large data sets? That's really our angle on all of this. Can you drill into the HNSW versus disk ANN and what those mean and what are their implications? So let me talk a little bit about sort of the origins and why everybody has been using these things. One of the amazing things about, as companies have gone and approached a lot of the build out on AI infrastructure has been that they've been using a lot of stuff that's been around for a while. And in fact, if you talk to folks, they'll be like, oh, well, you know, approximate nearest neighbor and so on.

5:31Like this has been around for a while. And in fact, there are better approaches that are coming down the pike on this. But the HNSW implementation that most of the databases, most of the vector databases out there ended up using, what came out of Lucene. And Lucene was one of the original, sorry, I really asked the search engine. But at this point, it's more of the search library, search infrastructure, index creation that a lot of folks use. And I'll definitely get this wrong. So actually, I won't name names. But other than just say most of the big names that you would have heard about and that we all talk about when we talk about vector databases, they all started with the code that was in the Lucene HNSW implementation.

6:11And in some cases, they may have ported it over to their language of choice because not all these things are written in the same code base. but they started with that. And that was a good starting point. Like if you were the fill in the blank, the original vector databases that were named in the OpenAI blog post that kicked off the whole vector database race, or the people who came out very quickly afterwards, it's a good starting point. The problem was that HNSW was not particularly optimized for disk IO. So what we found was that as we started to deal with the performance of this, and particularly with Cassandra that's a distributed database, we went and looked at sort of the structure and essentially the levels and edges of the graph that gets constructed.

6:58Because basically you end up breaking these things down. This is where the hierarchical part comes from. And then you end up having to do a partitioning, particularly where, you know, in a clustering, basically, this really comes home to roost when you're using a distributed database. Because when you look at the way something like Cassandra works, it actually goes and takes a query, vector non-vector, this is just the way Cassandra works, it takes that query, farms it out to a whole set of nodes. Like people who are running Cassandra have thousands of nodes. And so if you don't have something that first of all allows us to map that hierarchy of where the semantic position is of something in that cluster, and then further doesn't go and reduce the number of disk IO operations, you end up hitting a performance wall.

7:46And one of the dirty secrets of vector databases is that they all perform really good on small data sets because they effectively end up defaulting to what's in memory. And in fact, in the early days of when these vector databases were coming out, a lot of developers you'd read on Hacker News and elsewhere would just joke that like, for these demos people are doing, you can just do everything in memory on your laptop and you'd get better results. So that's where a lot of the stuff that we end up looking at for us, particularly for Cassandra's role in this stuff, we want to be the one that's doing really large data sets.

8:20For example, one of our demos, one of our rag demos that we use has the entire Wikipedia, Wikidata data set, and that's something like 500 million documents. And you can actually start to see the issues start to crop up in as few as 100 ,000 or so documents. But definitely when you get into the million plus documents, you start to see that these trade-offs actually have very tangible results in terms of relevancy, not just in terms of the performance, but actually starts to cause fairly significant drop-offs in relevancy, and you just start to get to junk back. And you see particularly where sort of all of the vector databases right now are sort of in an arms race with it, because really the business payoff on the other sort of important comment about vector databases to understand.

9:02And databases, I won't spend a lot of time talking about the database business today, but all the databases, whether you are us, whether you're Mongo, whether you're Pinecone, whether you're whatever, these are consumption-based businesses. And it means that we all love and we all do the hackathons and all that, but generally the businesses are built on having very large data set. If we can't vectorize that very large data set and make it possible to do like production rag at scale, it's going to be really hard for anybody to build a business around this. And so if you talk to anybody who's running a vector database, a lot of the focus is about how do we make it feasible and cost effective.

9:37There's a big cost dimension to this as well to do these really large ASINs. And that's why all of these choices, like we're really happy about disk ANN right now. We have a lot of stuff that shows it outperforms in real world, a lot of other options, but we're already going and looking at some of the stuff that some of the different approaches that will move beyond that. Some of them specifically to distribute a database space. disk ann is a an algorithm and approach for doing approximate nearest neighbor on disk presumably it's an infrastructure aware approach for approximate nearest neighbor exactly that's just principal difference there are a whole bunch of other pieces to it that again i would do too crappy a job of in terms of going into detail on but the principal biggest difference that we've seen has been we've been running this stuff as a service since early this year and so we've we've had the privilege to see a lot of the sort of real world, what happens when you do rag in a real world setting.

10:34And so when you start throwing a lot of IO operations, disk accesses, the problem, you start to see a lot of things that you wouldn't see otherwise. And you wouldn't see, for example, in a classic chat GPT scenario, because they're not, they do now because now chat GPT is actually does a lot of RAG as of a month or two ago, but you wouldn't have seen previously in sort of the conversational scenarios that we're really just throwing things at the model. And in as much as the database is involved, it's being used for conversation history, which is a part of RAG, but it's not where sort of the bread and butter of RAG really comes home.

11:12You made an interesting comment about the implication on not just performance, but relevancy as you scale vector databases. I'd love to hear you elaborate on that a little bit more. For a bit more context, I've shared here on the podcast and elsewhere, one of my observations is that it's really easy to get from kind of zero to a POC with RAG and with Dialog Agents based on LLMs, but getting from that POC to a system that you'd be willing to put in front of customers is like a lot harder. And one of the big elements of that gap is relevancy and all the details that go into like embeddings and constructing the context for the LLMs.

11:57But at the heart, like it's a relevancy challenge. And as a corollary to that, the folks that I've run across that have deep experience in search and getting search results tuned up seem to kind of get this and know how to fix this problem. I'm wondering, A, does that resonate with you? And then B, like talk a little bit more about relevance and kind of the way you see that playing out in vector databases. Yeah. So there's a couple of different pieces to it. And in fact, you're going to start to see a lot more of this just in general as people talk about. You have, you know, your precision, your recall, your accuracy.

12:35You've got ways of measuring those. You've got things like F1 that combine precision and recall because they're trade-offs. And so as we look at those, these fluctuate over time, and they fluctuate as a function of the data set that they're operating under. So generally, we go and we look at stuff, and most of your interactions, so it's probably worth sort of, there's two ways of looking at this. You can look at the vector data. Remember, most of the action around vector databases prior to this year was in using the vector databases as search engines. If you go back in time, take a look at what, for example, like the names we all throw around today, when you're looking at essentially what were these companies doing?

13:17Folks like Chroma, folks like Weviate, all that. Like a year ago, they were primarily looking at being search engines. And the idea was that, and as you just pointed out from search concept, that a lot of this stuff was rooted in search. And the idea was that keyword search, which is what a lot of folks are most familiar with, has a bunch of limitations. It actually works well enough in most situations, but generally people want it to move from a keyword to a semantic approach. And so then that's where doing a vector search comes into play. And by the way, you can actually do really good recall with no LLM really in the loop.

13:56It's more of an LM rather than LLM. Nowadays, we just call it an embedding model that is just for the purpose of going and reducing your text input into a vector that I can now go and do a similar search. So now we get into this stuff and we're putting in much larger data sets and we want to go and try to figure out how do we go and how do we measure, for example, how accurate are these results? Basically, are we getting a lot of false positives when I go and do a search? Am I getting things back? And between accuracy and precision, we narrow down sort of we get a good sense of sort of we're able to calibrate the sort of the false positives.

14:34And then we also get into sort of the recall pieces of it, which translate down into like things that should have come back. So it's actually sort of more of the false negatives. And again, this is very overly reductive, but we start to get into that. Again, you can measure these things and you can measure them. And then again, I also mentioned F1, which combines basically you have a trade-off from these and from recall to be able to see. So people have created other measurements around these. And again, this is, if you go and look sort of over the next six months, just for context, This is where a lot of action is going to be.

15:05You're going to go to, if you go to any one of these, it's not there yet, but I can predict within a month or two. There you go. There's my prediction for the new year, which is when you go to a vector database homepage, what you're going to see is for any of these open source or commercial, you're going to see all these great graphs that show number of documents and what precision recall F1 score and all of that are. Because that's where, like when you get past, you know, you mentioned sort of the demos, It's like past sort of going and loading a bunch of this stuff in. That's where the action becomes.

15:35Now, that said, apart from just the raw database capability of doing these results, there's a big piece around whether actually a big data prep piece that goes directly into it. And that's one of the things because you'll hear people say, oh, but real world this and on there. Right. What they're saying is like, it is garbage in, garbage out. And so that's where we get into things like chunking and so on, where I go and I take this piece of content and I need to break it into a set of things, not just from the fact that you want to get the appropriate piece from a storage standpoint, but more importantly, when I chunk these things, we take these documents, depending on how I chunk them, I could lose.

16:17And chunking is exactly as the name implies. Taking this, one might imagine, a large PDF and breaking it into bite-sized pieces. I can lose a lot of context that the smartest fellow alum in the world is not going to be able to put back if I happen to break things apart, you know, if I did it naively or whether I did it with a better understanding of the structure of the document. So chunking itself, actually, in terms of putting the data in is where you see a lot of work. And if you, for example, you know, we look at some of the frameworks people use for the stuff, Langchain and LOM Index, they put a lot of effort into going at the ingestion stage.

16:51And again, that's part of the reason why Lama Index is named what it is. And actually, as an aside, sort of you get to the differences between those two projects and the philosophies behind them. Langchain, as the name would imply, is about chaining LLM invocations. Lama Index does that. But in order to do that, Langchain does do a bunch of ingestion stuff. Lama Index, as the name implies, was really about getting that content and building those indexes. Of course, they also do orchestration. But you see when you go and look at the folks doing those projects and, you know, whether you're talking about Harrison or Jerry, like you can sort of tell they both had sort of a difference problem they were trying to solve.

17:27And again, when we sit down, we have to think when to build like a rag app, we got to think about the intent. They both become big problems. And depending on when you want to go and improve the accuracy of your results from the vector database, some people when correctly will go and say, oh, focus on getting the data in there, because that's going to have a big impact on your scoring. And other folks come in and say, well, that's true, but a big other piece of it is kind of how we break apart and construct the context that we're using to get to generate the vectors that we want to look up on and then what post filtering we do.

18:01And the answer is both of those are very important. Do you think since you offer it up predictions, do you think we'll get to a point where this all happens automatically kind of in the infrastructure and the user just needs to kind of bring their documents and get great results? Or will there always be a degree of fine tuning and tweaking that needs to happen in order to get desirable results out of a RAG type system? It's a good question. And as we tease that apart, we're actually going to get into a bunch of the subjects that I know you wanted to talk about. So what's our holy grail? Our holy grail is that the process of getting my data and using it within a Gen AI conversational experience.

18:49And as we've seen, conversational doesn't mean chat anymore because we've got multimodal and we've got drawings and pictures and actually dynamically generating a graphical user interface is something that we can do now. But I think probably you and all of your listeners probably already know that, which is that Gen AI doesn't just mean chat anymore. It just happens to be the simplest way to prototype. The first really important piece is, and this is not self-evident to a lot of folks. In fact, actually a lot of folks in the AI research domain don't fully get this, which is that real world AI, particularly like business use cases, needs you to be able to bring your own data.

19:24And that data, and again, this is really important, is not static corpus of data. What people are trying to do, and this is why people are using RAG, you're talking about data that is live data that is changing, that is oftentimes maybe confidential or proprietary, things like electronic medical records or people's financial statements and stuff. That is never going to get fine-tuned into a model. It is never going into the model, which means it's always going to go into infrastructure that's around the model. So once we do that, we're talking about RAG or some variant of RAG. If you looked earlier this year, there was this charmingly naive debate on RAG versus fine tuning.

20:06And seriously, you had really smart people getting into this debate and saying things like, oh, fine tuning, we won't need RAG as a fine tuning. And I was like, you're never going to train a model on people's electronic medical records or bank statements. If you want models leaking personal information, that's how you get models leaking personal information. right but that was it was the intersection of the fact that you had a lot of people were just very focused on building these really cool models and hadn't sort of zoomed out and said how are we going to use this stuff in more applied ways and so this year has been about the collision course of research and applied in the ai domain in a really exciting way but you see it playing out in like a day by day so we do know that but then the question becomes okay what do i have to do to get my data into a way that i can effectively retrieve it and that's where you've got the frameworks, and I mentioned a few of them, and by the way, there's a whole bunch more, but the kings at this point are Langchain and Lama Index, who have just been moving really, really fast on this.

21:02And you get a lot of people complaining about the code bases there, like, man, like, they're just, I'm like, yeah, but these guys are following the stack. You got to admire that, because that's what that's all about. But as a consequence, there is a lot of trial and error. So I see a lot of rag projects. And so going back to what you're saying, like, is it just going to be, somebody's just gonna make this dead simple. Well, yeah, ultimately there aren't a lot of people working on it. Right now, when you go and do a rag project, you spend a whole bunch of time. And it's not dissimilar, I think.

21:36So in that way, there's a lot of ways that an AI project this year, a Gen AI project is similar to the ML projects that you and I have been looking at and involved in in previous years, right? And then there's a lot of ways that they're completely different. And the ways that they're very similar is that for all the talk you're talking about, about like the cool stuff, like the day to day is really a lot of data engineering, which is a euphemism for just like data cleansing and a whole bunch of. And this was again, this is why people are using Python and such. Just a whole bunch of data munging just to get that data from the state that people have it in into a way that's going to yield the best results.

22:15So is that going to go away? Yeah, absolutely. Absolutely. And people are going to put a lot of wrappers around what currently requires you to do a lot of spaghetti architecture. And a lot of folks will, you know, and you'll see a whole bunch of companies that focus just on that because you've got a whole bunch of data in the systems, in the databases that people are already using. They're using streaming architectures, whether using things like Kafka or whether using their databases, relational databases, whether using non-relational databases. Like all that data needs to go into this stuff. And then you've got more importantly, like that's all your structured data.

22:51You've got a lot where AI starts to really blow things away is with unstructured data. And the unstructured data in most cases, you know, again, I spoke to this earlier, the hello world that for most RAG apps or RAG frameworks and most vector databases is chat with PDF. And the piece of that is, again, how do we take apart that PDF, which is at this point is now the canonical piece of unstructured data that everybody tests with? Not the most interesting one, but the most pervasive one. And then we go and say, OK, how do we take it apart? And then you start to get really into the world of mundane data at that point.

23:29It's like, is this an insurance claim? Is it a legal contract? Is it a research report? Is it a whatever? And every one of those has a set of heuristics involved that may be actually defined procedurally. Because, for example, in the case of a research report, or excuse me, a legal document, you don't actually need like AI to take it apart. Like every contract has exactly the same structure. So you can do a bunch of string munging and extract it and turn chunk it, which people do. There are entire companies and startups that do that. Or you can do things more cleverly, which is to go and you can have an ingestion loop where you're feeding this stuff into the LLM, having the LLM guide the ingestion flow.

24:08So you have the LLM supervising the dismantling of this unstructured content into chunks that then later at rag time, you're then going to go and be able to grab out. Right. So again, this is and I'm sure because you've got a lot of folks, a lot of your your listeners and stuff are probably in the middle of rag hell right now. And so they're probably like hopefully nodding like, yeah, no, I just did that this week. All that stuff has to go away. But part of the problem becomes it's like everybody's learning as we go along. And so like you don't want to prematurely optimize and automate for what the use case was last week.

24:44that was just the Hello World app when what we try to do next week is something more complicated. Because every time somebody brings in a new data set or a new type of data, it's like, okay, can we reuse the approach we use last time? Can we abstract on it? Can we build a new framework? And hence, there's a new BRAG framework every week, right? That's a fair and interesting point. Like you could over-optimize on PDF retrieval and get to anchor down into that and totally miss multimodal, for example, is what you're saying. Like there's a danger to premature optimization, which is kind of a truism in software, right?

25:22No, I mean, your example exactly was right. So again, most of what people are doing right now is smart knowledge bases, which by the way, not a bad thing. Like a lot of use cases, a lot of practical applications, a lot of happy users and a lot of businesses are gonna save a lot of money and make a lot of money just having AI knowledge bases. And for the knowledge bases, a lot of that stuff, the content that they're sourcing from are support contracts, things like that, or whatever documents in PDF, right? But like multimodal, the stuff you see from multimodal that people are building now, because we now have multimodal, obviously we have multimodal in GPT-4, but you actually have multimodal now, even in the open models.

26:04That's like the next phase of magic, right? To be a little hand wavy, but like that is also rag based or can be, but it has a completely different ingestion flow. It has actually two pieces of ingestion flow, because a lot of this stuff that we're talking about here is data prep time. But in multimodal, you have a real-time ingestion flow. You see all the examples where people, I'm drawing a blank right now, the name of the draw program that everybody's using to do the, you draw the thing and then you circle it. A sketch thing? Yeah, the sketch thing. Right. You know what I'm talking about. Yes.

26:34So the magic on all that, on that particular use case, is that they built a plugin for, it's basically an ingestion plugin, right it's like selecting the piece of the drawing area that it's then they've gone and wired in and sends it up to gpt4 right so that is a nice hello world example the minute i start applying that in to different use cases now i'm going to be like okay how do i capture is it something from my screen is it something from this app oh i'm taking something from you know so there's going to be a whole bunch of software engineering that goes into into that we've talked a lot about rag I guess, you know, a question for you that I've been grappling with a little bit is, you know, as a vector database is kind of this vector capability.