#153 Ed Anuff: Unpacking AI's Role in Data Management

0:00I can also feed it the facts at the time rather than having it deal with its foggy memory So if I can give it the set of facts, then I can eliminate hallucinations as well. So those combination of things, it's basically if I can have the LLM, either I can tell it your preference is the information that I'm supplying to you. And if you genuinely don't know, say you don't know. That combination ends up with very low hallucination output. Hi, this episode is sponsored by Salonis, the global leader in process mining. AI has landed and enterprises are adapting, giving customers slick experiences and the technology to deliver.

0:46The road feels long, but you're closer than you think. You see, your business processes run through many systems, creating data at every step. Solonis reconstructs this data to generate process intelligence, a common business language. With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Solonis, enterprises get faster, more accurate insights, a new level of automation, and a step change in productivity, performance, and customer satisfaction. Process intelligence is the missing piece in the AI-enabled tech stack.

1:32Search Celonis, C-E-L-O-N-I-S, to find out more. Hi, I'm Craig Smith, and this is Eye on AI. This week, I speak to Ed Anuf, Chief Product Officer at DataStax, about vector databases and their role in AI applications. We explore how retrieving trusted data into large language models addresses the hallucination problem. Ed explains the growth and demand for vector databases and key factors like production costs and benchmarking relevancy. I hope you find the conversation as informative as I did. I have been over at DataStax for the last close to four years now, a couple of months short of that. Prior to that, I was at Google for about four years and joined Google when it acquired Apigee, which was the API management company that I was part of for a number of years.

2:47I've worked in a variety of companies, enterprise software, as well as things like blogging software and, you know, a few consumer things as well. So I've been in tech for a long time now. Originally an RPI graduate, Rensselaer Polytechnic Institute, way back in the day. Yeah, yeah. And what made the move to data stacks? And I'm curious, I mean, vector databases in particular are all the rage right now because of the hallucination problem with LLMs. But what was data stacks doing before the GPTs came along? Well, so we make the Cassandra database. It's a very popular open source database that is designed for scale out data.

3:42And Cassandra is the database that's used by Apple, by Netflix, by Federal Express, by any of a large number of companies that need to deal with scale-out data that they're distributing to users on a global basis. So part of the attraction of coming to DataStax was the fact that it was dealing with very large data sets. And the goal was to make that data available for AI and machine learning use cases. And that was something that we were already well underway on. a number of our customers were well underway on. Obviously, some of the users that I mentioned are very well known for their use of predictive AI.

4:36For example, Uber as well also uses Cassandra. It's one of their primary databases, and they've published a lot of great research papers and done a lot of talks about how a lot of the predictive AI that powers the Uber application is built on top of Cassandra. So our goal was always to bring that to the customers, as part of the product. And then in the middle of that, generative AI hit and obviously has really sent everything into overdrive. Yeah. Cassandra, what kind of a database is that? So it's a NoSQL database. It is similar to databases like Mongo and Couchbase or DynamoDB or some of these others, which means that it, You know, although it does let you use a query language like SQL, standard query language, which is what most relational databases use.

5:33But it's more designed for situations where you're dealing with large amounts of data and the data schema change can change or can be very flexible at runtime. So you find a lot of the, these types of databases became very popular for the large internet services. Google, Facebook, Amazon were really the pioneers of this type of database. And Cassandra itself spun out of a project at Facebook and the technology was open sourced by Facebook. So that's where you tend to see this type of technology is where people are dealing with large amounts of data that they're using in their interaction with their end users.

6:18And the, I mean, I've got a few questions.

6:25So DataStax builds itself as a real-time data company, a real-time database service company. or I guess you build on-prem as well. But is that exclusively for now that we're dealing with Gen.AI for vector databases? How different are vector databases from Cassandra? And why this sudden shift that I'm seeing to vector databases from other forms of storage? So a couple of different questions there. So first, we do talk about ourselves as a real-time database. We do so in the context of AI as well. And the reason is because Cassandra is designed for very high throughputs of data. You can write to the database very quickly, but more importantly, that data is immediately available for reading as well.

7:25And it turns out that becomes really important in Gen AI, and I'll talk about that in a minute. But along the way, what we saw, and basically starting last year and going into this year, is that a new application pattern emerged, and it was inspired by ChatGPT. And I think we've all experienced ChatGPT, So it's useful to sort of start with that because it helps explain why vector databases are important. When you show up at ChatGPT, part of what makes ChatGPT interesting is not just the question and then you get a response. It's actually the chat part of it. So you ask a question, it gives you a response, you ask it for some elaboration, or perhaps you're having it help write you some code.

8:17You'll go and say, oh, that isn't exactly right. Could you make it do this? And so it has a sense of history. And if you know about how LLMs, large language models work, they're completely stateless. The model itself, when you ask the model a question, it doesn't change. That model is frozen. It can later be tuned or retrained, but the model itself, the data comes in, it actually can be represented from a software development standpoint. It's a very simple black box. I have an input, which is often called the context or the prompt, which is a form of context. And then it has a response. So where do things like the history come from?

9:00Where do, you know, those actually come from a database. And so ChatGPT has a database that sits next to GPT-4, and it knows the history, it captures the response, and it's able to create a personalized conversation with you. And that's sort of where the magic happened. And then OpenAI, the company behind ChatGPT, did a blog post in December of last year. Because at the time, they weren't fully in the ChatGPT business. They actually had launched us an example. What they were really trying to do was get people to use their models. They wanted you to use GPT-3. They want you to use GPT-4. And so they said, if you like this thing that we've built and you wanted to build your own, what you need is a database that sits well with the large language model.

9:50And oh, yeah, part of what helps that database sit with the large language model is if it is able to store and query vectors. is the vector is the numerical representation of essentially the concept. What it does is it takes that whole string of text and it reduces it down to a very, well, I say reduce, but in some cases actually it can be much larger, a very long multidimensional number that represents what that word or phrase or sentence or paragraph reduces it down into a context and and or concept concept semantic concept and that's how llms think they don't really literally think but but they they represent these these you know what we consider ideas or concepts they represent it in a large multi-dimensional space and it's the distance between things in that space that says how similar or different they are.

10:54So this is an important capability that lets a database work well with LLM. Now, there are some other uses for vectors. Some databases have been supporting vectors for other reasons, basically for doing better search results for a while. And some of those were among the first crop of databases that people started using with LLMs. um all the other database vendors very quickly followed suit by by this time we're almost a year since you know a couple of months short of a year since since opening i wrote that blog post that kicked off this whole thing and at this point most major databases have some some vector facility because it becomes really important there's lots of people who are building these types of applications in enterprises and startups and so on and and they need databases and they need a database that can work with these models and that's why you're seeing everybody talking about vector databases sorry that was a little bit of a long response no that's uh that's wonderful that really clarifies it quite a bit so for data stacks uh at this point when did you introduce vector databases?

12:10And is your business, you know, 6040 with traditional relational databases and now vector databases? Where is the future of the database business going from your point of view? So really good question. So we introduced the vector capability, you know, since we're built on top of open source, we were able to get the code out there very quickly early in the year. We had it live in our cloud service in April of this year. And we're at a point right now where 40 % of new signups, people coming to our cloud service, are doing so for vector usage. Actually, let me amend that, actually. it's actually more than half are using it for vector usage.

13:09And about 40 % actually have never used Cassandra before. So there are completely new users who came to the service because they were looking for vector databases. So this is pretty important for us. Now, I suspect that some of the other database companies are seeing similar stuff. I'd love to say, oh, we're uniquely suited. I do think we've got some great important capabilities that others don't have. But I also do, I think that the bigger point is that there's a major catalyst this year with people trying to build these types of applications, and it's causing them to go and rethink what databases they're looking at and evaluate different databases and so on.

14:01So long and short of it is that's why, you know, if you're following the database industry, the entire conversation is around vector databases because new application types don't come out that often. The last time we had a new application type of this level was probably mobile. and that catalyzed a whole bunch of stuff. And then prior to that was probably e-commerce and websites and the web itself. And you start to get that hyperbole as to the, but I've been around for a while. I'm sure you've been following this stuff for a while. We all have seen this thing where there's a new application type and companies rush to respond to it and it really changes everything.

14:53And, you know, within the database space, again, probably the web. And then prior to that, probably client server being like the biggest catalyst. So very important to us, long and short of it. Yeah. Can we back up just a little bit? Because I want to talk about some of the products. But I have questions about data that I've never satisfied myself with answering. You know, I was close for a while to a company called LabelBox that is a platform for labeling data. And it fascinated me. This is, you know, during the supervised learning when that was the dominant technology in AI. And in reading, and understand my background is not as a technologist.

15:54I was a foreign correspondent covering politics and, you know, conflicts and stuff like that. So the amount of data that's being produced is increasing, it seems, exponentially. I don't have the numbers in front of me. Uh, and, uh, so you, you, when you talk about real time data, it's, it's, uh, it's, you know, writing and reading data on, uh, in real time, but presumably some of that data then gets stored for, uh, for long-term, uh, usage or retrieval that it's not being retrieved, uh, frequently, but you don't want to lose it. And where does all this data go eventually? Because it's not being erased.

16:49And at the time that I was looking at this, there's kind of a hierarchy where it gets moved from one medium to another until it's finally on literally tape drives stored in a mine shaft somewhere. But do you understand that? Where does this data go? And is it being accumulated to the point that someday there's going to be more data than any other artifact in the world? It probably already is. long and short of it is um it's somewhat somewhat depends on the use case and it really ends up um there there's two pieces to it there's there's sort of cost and policy um and um depending on the systems and depending on the so first of all you're correcting your basic statement which is particularly when we talk about real time um you're capturing data you are You're perhaps doing analysis over ranges of it.

18:00You're keeping it around. And you are accumulating it. And a lot of companies do accumulate it. And the question then becomes, how long do I keep it for? In some cases, depending on what you're doing, the cost of getting rid of it might be less than the cost of keeping it. The cost of storing data has dropped. over the years. And so what ended up happening was a lot of companies, particularly in the internet age, just kept on accumulating that data. Because again, querying and deleting a range of data is actually from a computation standpoint, from the process of doing that, be just as expensive as writing that data.

18:48So it could be much more actually. And then the cost of storing the data can be very low. So then you see a lot of companies just go and say, okay, we're just going to keep this data. just because it's not like a, it's not as much as a business decision. It's more just like, okay, the cost of doing something, we'll get around to it. And maybe they do it on certain basis. Then you have sort of the policy questions. And the policy questions come from either business or regulatory. And so then you have the questions of, in some cases, some policies require you to retain the data for a period of time.

19:32Other policies go and say, you shouldn't retain it. Many people on like corporate email systems and so on, things are auto-deleted after 30 or 60 days, right? For as a company policy, because if that's your standard policy, then you don't have the data, you know, then the data isn't there and so on. And it may not be desirable to keep it for various reasons. So that becomes a whole decision point in and of itself. And then there's security reasons as to why you don't want to keep data around for a long period of time as well. And so, for example, companies that are very concerned with privacy actually make it and advertise it.

20:11They advertise that they don't retain any data, personal data whatsoever. And so there's a whole set of things around that. And so the long and short of it ends up being that you probably could look at 100 different companies with their data retention strategies and find 100 different answers. Because a lot of it will have to do with the quantity that's being accumulated and the cost associated with it. But the companies that, and the ones that we talk about, for example, like financial statements and things like that, that only get, you know, updated and aggregate, you know, on a monthly basis.

20:56That's why you can get, for example, a bank statement. They increasingly made it becomes harder to do because they shift it from to what's essentially called cold storage or less accessible forms of data. But they still keep it around for 20 years in some cases if you need to get something. Whereas, you know, the internet companies that are capturing and literally storing a row in the database every time you load a page view because they're doing it for personalization and ad tracking purposes, those, they will collapse and compress and create summaries and delete the rest. And the interesting thing is that those summaries end up getting created through MLPRO.

21:47That actually ends up being an important usage of machine learning is to go and condense those 10 ,000 clicks into something that they can then just store and they can delete the records of the previous 10 ,000 clicks. So it is a whole thing in and of itself. I don't think there's sort of a one-size-fits-all strategy on that. It ends up, as I said, ends up being with how much data you're dealing with and how quickly it's changing. Yeah. And so with data stacks, is there a drop-off then that a customer can set that it's reading and writing, but after a certain period of time, it goes into some sort of a storage bin that's a little more difficult to access?

22:39How does that work? Well, so that you definitely can do that. There are a variety of mechanisms too, but it actually gives me an opportunity to talk about one of the cool features, which is we do actually have the ability for at the individual role level in the database, and it's pretty unique to Cassandra that you can make the data expire. And as a consequence, you have lots of users of Cassandra that take advantage of that. And they just they're like, OK, I only care about this data for a month or whatever. And then after that, I don't want to have to go through the problem of crawling through all my data and checking each row and saying, oh, is it older than this amount of time?

23:23Let me delete it or whatever. Rather than that, the database just automatically does it. It just drops it off of the database. And so then you have this data set that always contains your last 30 days or whatever the period of time that you want it to be. So that's one that is that. And why that becomes important is that, again, where you get into these situations is where you have these very large sets of data. And it is one of the areas where we play and where Cassandra is uniquely suited is for the extremely large data sets. Yeah. My familiarity with vector databases comes from addressing the hallucination problem with LLMs.

24:08And a lot of companies are building a vector database. Even after they tune an LLM, they still have this hallucination problem. So they build a vector database with trusted data and the LLM becomes simply a language interface. Is that how DataStax customers are using your vector database? Yeah. So that is a description of a process called retrieval augmented generation, or we often hear it called RAG, right? So if you're at a conference, an AI conference or something, everybody will be like, oh, RAG, RAG, RAG. and you're like, what is this? And so it is this process of, and as the name implies, retrieval augmented generation, goes and says that the generation is, that's the name would apply, being augmented by retrieval from data sources that are fed into the LLM at the point of inference.

25:07And in fact, what you can do is tell the LLM that it should only consider the knowledge that is supplied to it at the point of inference. In some cases, it's just to supplement it. But if you really want to eliminate hallucinations, you just go and say, look at, use your reasoning powers, but don't use your knowledge. And the reason is because one of the aspects of hallucinations is there's a lot of causes of hallucination, but one of them is that that LLMs have a foggy memory, if you will. Right. Just like humans. They although perhaps a different mechanism, although not that different a mechanism, but that would be an entirely different conversation.

25:50The, you know, so rather than it going and trying to, you know, remember exactly, you're like, here's the set of relevant content, choose among them and choose among these things and give me an output. And you do that for two reasons. As you said, it's trusted. But there's also another important piece of that, which might be that it's also, you know, sensitive. So, for example, you could have an LLM that is providing you with medical information, medical, you know, medical recommendation. And you don't want the language model fine-tuned on a set of electronic medical records. The LLM is not good with private information.

26:45Generally speaking, anything that goes into the LLM, it's going to leak out. Yeah. Either inadvertently or advertently. There's no way that you put access control on the knowledge that an LLM has. You could do something convoluted, which I see some people say is, oh, I'm going to have a second LLM spill the beans and filter it out. But now you're getting into this Rube Goldberg architectures. It's much better to just supply the LLM with that sensitive information as it needs it. Remember, an LLM has no memory in and of itself. So I can say, here's Ed asking a health care question and and here's his electronic medical record.

27:30And LLM looks at that, goes through and says, you know, well, you know, Ed, you know, probably, you know, need to exercise more because, you know, as I, you know, you know, your weight has gone up in, you know, over the last six months. Right. But it doesn't remember that fact and then later be in a conversation with Craig and say, hey, Craig, you know, you want to exercise more because you don't want to end up like Ed, right? Like, I don't want, I don't want Vell L.M. to know anything about Ed when he's talking to Craig. And I don't want it to know anything about Craig when he's talking to Ed, right?

28:04So that's an other piece of it. Now, the other aspect of it is, again, I can also feed it the facts at the time rather than having it deal with its foggy memory. So if I can give it the set of facts, then I can eliminate the hallucinations as well. So those combination of things, it's basically, if I can, you know, if I can have the LLM, either I can tell it, your preference is the information that I'm supplying to you. And if you genuinely don't know, say you don't know, that combination ends up with a very low, low hallucination output. A lot of these things are more art than science, which is frustrating to many as they're building these systems.

28:52And you can make it more exact. Again, we've done this. We do this for we have an AI co-pilot that helps you use our products. And what we've done is we supply content out of our documentation into it and we tell it you exclusively use that. and then it gives a result that doesn't have any hallucinations. Now, we also do a little bit of fine-tuning as well, but if you really want to eliminate the hallucinations, it's a good mechanism. Yeah, and right now, I know you have a product called AstraDB. What is your primary product and what are the use cases that people are turning to it for? Sure. So we have a cloud product called AstraDB, and we also have a self-managed software that people can run themselves called DataStax Enterprise.

29:53And they're both built on top of the Cassandra open source database. You know, our fastest growing product is the cloud product, as one can imagine. We do have many, many enterprises, actually. The majority of the Fortune 500 is using DataStax Enterprise, running it in their own data center. Both of those groups are very, very interested in the vector database capabilities. I talk to, so it isn't just the ones in cloud. I talk to customers every day that are doing things with their sensitive data that they need to keep in their own data center. But yeah, most people who are new to our product and our company do come to the cloud service.

30:47They come to AstroDB, they do a sign up and they're just right within a couple of minutes there that they are able to create a database and connect it to their applications. And the type of applications that people build are all over the place in terms of applications that are powering mobile apps, applications that are powering websites, applications that are being used for things that are controlling, Internet of Things devices. And as I started out by saying, at this point, for brand new people coming to the service, the number one catalyst is generative AI. Maybe that'll even out or slow down over time.

31:38but but i do think that just i think it just has to do with the number of new applications people are building in general that that a lot of that focus is ai related right now yeah uh the um does does the suite of products include the the vectorization of of uh data because uh you know in order to put it in a vector database they need to be vectors uh so that's that's a really good question. So what we do is we actually do have the capability within the products for being able to use open source models. The selection of the model that you use for the vectorization is actually a really important decision.

32:24Some people want to use OpenAI. Some people want to use Google's models. Some people want to go and use open source models. the different models all have different costs associated with it. And as a consequence, you see that this selection process, people will use a smaller model for vectorization because it's much cheaper. So we enable all of that. As I said, we do provide an open source model where that makes sense. but majority of people are using, you know, they're, they're, they're using something like open AI or they're using one of the Google models. Those tend to be the preferences. Yeah.

33:12But on your platform, you have those available or does someone have to? Well, there's, yeah. I mean, we, we, we let you integrate your account. You can go and put in your, your open AI account. Yeah. And I've been writing recently about the cost of inference and rate limits on large language models, which are constraints for enterprises. Does using a vector database lower the cost of inference and does it reduce the number of tokens that you're putting into or pulling out of an LLM because you've got the data vectorized already someplace else? Yes. So it depends on what it is that you're trying to do.