Open source, on-disk vector search with LanceDB

0:28Welcome to Practical AI. and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I am CEO and founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing good today. How's it going, Daniel? Oh, it's going great. I was just, well, we were just remarking before actually starting the recording that one of the great things about doing these episodes is that we get the excuse to bring on the show the coolest open source and tooling and other projects that I'm using day to day and get the chance to interact with.

1:22And one of those is LanceDB. And we're really excited today to have with us Chung Hsu, who is the CEO and co-founder at LanceDB. Welcome. Thanks. Hey, guys. Super excited to be here. Thanks for having me on. Yeah, yeah. Well, first off, congrats on all your success. I was scrolling through LinkedIn and saw a video of LanceDB up on the NASDAQ screen in Times Square. So that was cool to see. That must mean good things, I'm assuming. Yeah, this is possible via Brex and also Essence VC. So big thanks goes out to them. Cool, cool. Well, yeah, well, I mentioned I've had a chance to look through some of what you're doing and actually use it day to day.

2:11Actually, that was a result of a previous episode that was, I think, titled Vector Databases Beyond the Hype with Prashant. I think the question that we asked him was like, oh, there's all these vector databases. You've compared all of them. what are some of the things that stand out or some of the vector databases that stand out in terms of what they're doing technically or how they're approaching things. And one of them he called out was LanceDB. I think in particular, he was talking about kind of on disk index stuff. And so I'm sure we'll get into that and a little bit more, but that's how I got into it.

2:49So I recommend listeners maybe go back and get some context from that episode. But as we get into things, could you maybe give us a little bit of a picture as to how LanceDB came about? I know there's a lot of hyped vector database stuff out there and people might not sort of realize how these things were developed, how they came about, what the motivation was. And so if you could just give us a little bit of a sense of that, at least for LanceDB. Yeah, absolutely. And first, I wanted to also give a big shout out to Prashant as well. As you were saying, there's a lot of hype and noise in this area.

3:27There are a lot of different choices. And for users and developers who are building generative AI tooling and applications, it's always kind of confusing, like, which one is good? And should you listen to the marketing from one tool versus another? So it's great to see someone with an engineering background who can write so well to actually take the time and just try out a ton of different tools and interview a bunch of different companies and come to his own conclusions. I'm super happy and excited that he's a fan of LandCV, and we hope to make that better for him and also all of our users. So back to LandCV, I think, so we started the company two years ago at this point, and we didn't start out as a vector database company, actually, because I think if you kind of remember, ChatGPT is barely one year old.

4:20Yeah, the dawn of AI. Yes, exactly. And so the original motivation was actually serving companies building computer vision and building new data infrastructure for a computer vision. So I had been working in this space for a long time. I'd been building data and machine learning tooling for about almost two decades at this point. I started out my career as a financial quant, and then I became involved in Python Open Source. I was one of the original co-authors of the Pandas Library. And that really got me sort of excited about Open Source, about Python and building tools for data scientists and machine learning engineers.

4:57And so at the time, this was in 2020 and 2021, what I observed was that the company I was working for, TubiTV, it was a streaming company. So we dealt with both machine learning problems for tabular data and also for unstructured data like images and video assets and things like that. And what I had noticed was that anytime a project touched this multimodal data for AI, from images to the text for, let's say, subtitles or summaries to the poster images, these projects always took a lot longer. They were much harder to maintain, and it was difficult to actually put into production. At the same time, so my co-founder, Leigh, who I had met during my days at Cloudera, he was working at Cruise and sort of dealing with the same issues.

5:51And so we put our heads together, and our conclusion was that, hey, it's not the sort of top application or workflow layer or orchestration layer that's the problem. It's the underlying data infrastructure. If you look at sort of what's been out there, like Parquet and Org has been around, and they've been great for tabular data, but they're really – really suck for managing unstructured data. And so we essentially said, hey, what would it take to build a single source of truth where we can toss in the tabular data plus the unstructured data and give much better performance at a much lower cost, a total cost of ownership, an easier foundation to build on top of for companies dealing with a lot of vision data.

6:38And so this comes in handy when you want to explore your large vision datasets for, let's say, all time is driving. This comes in really handy for things like recommender systems and things like that. So we started out building out that layer, that storage layer in the open source. And that took about a year's worth of effort to really get to a shape that is usable, kind of like Parquet or Oric and other formats in these tools. And that was when generative AI became really, it burst onto the scene and became sort of a revolutionary technology. And what happened at the time was we had originally built in a vector index for our computer vision users to say, hey, let's deduplicate a bunch of images or let's find the most relevant samples for training, for active learning and things like that.

7:31And it was sort of that open source community that discovered that, hey, this can be really good for genitive AI as well. That's when we sort of separated out another repo to say, hey, this is a vector database. And it's much easier to communicate with the community than to say, hey, you're looking for a vector search. Use this columnar format. And so that's how we got on to this path. Quick question for you. It's really a follow-up to something you said. It's been a couple of moments now as we were going through that. But I was just curious, when you were talking about kind of going through the analysis on the top workflow versus whether it was infrastructure, and you said y 'all concluded infrastructure.

8:12I was just wondering, you kind of went on past that into that, but I was kind of wondering, how did y 'all come to that determination? For those of us who are not deeply into that thought process, I was wondering where your head was at when you were doing that. Yeah, it wasn't an easy decision or a conclusion. thinking back it was kind of you know so it was like 2022 initially seemed pretty crazy when we sort of first came up on it right if you think about it it's like why would you make a new data format like in 2022 like Parquet has been working so well and I think it was really observing the pain of our own teams and also we went out and interviewed a lot of folks managing unstructured data.

8:53And so for them, it was, you know, one, data was split into many different places, like the metadata might be managed in Parquet, and the raw assets are just dumped onto local hard drives or S3. And then you might have, you know, other tabular data managed in other systems. And they, they would always talk about how painful it is to stitch everything together, and manage it all together. And some of the outcomes are like, it's really hard to maintain those data sets in production. Like you have a Parquet data set that has the metadata and then links to S3 or something like that to all the images.

9:29And then somebody moves that S3 directory or something like that. And now all of your data sets are broken. Or something like we would interview folks are like, hey, what are you doing to explore your visual data sets and things like that? And they're like, well, you know, I use MacBook and there's this app on that called Finder. And if you single click on a folder, it shows you a bunch of thumbnails. It was sort of this horrible way to actually work with your data. But it was because it was so hard to manage all of that, that machine learning engineers and researchers were stuck with these subpar tools.

10:04You mentioned kind of this transition of thinking from some of the original use cases that you were talking about with computer vision to this world of generative AI that we're living in now. From my impression, from an outsider's perspective, it seems like LanceDB has kind of positioned itself very well to serve this kind of generative AI use cases, which I'm sure we'll talk about in a lot more detail later on. I'm wondering, from your perspective, how has that overwhelming demand for the generative AI use case kind of changed your mindset and direction as a company and a project and open source tooling and all of that?

10:47and how do you envision what you're targeting as the use cases moving forward, I guess? I think certainly generative AI has brought in a lot of different changes and new thinking. One was the sort of focus around use cases of semantic search and just retrieval in general. I think with the advent of generative AI, I think retrieval becomes much more important and a new book could us. For us, what that means is, you know, increased investments in terms of getting the index to work really well and really scalable, and then sort of making that data management piece to work really well as well and integrating with frameworks for RAG and for, you know, agents and for just generative AI in particular.

11:40When we started out, inevitably, we were dealing with multi-terabyte to petabyte scale vision data sets and things like that. And we're still dealing with a lot of that. But for generative AI, I think there was a renewed focus on ease of use because a lot of users are coming in who don't have years of experience in data engineering or machine learning engineering. And what they're looking for is an easy-to-use an easy-to-install package that doesn't require you to be an expert in any of these underlying technologies. We also spent some effort into, okay, that was sort of the motivation behind us making LansDB, the vector database, one, open source, and two, embedded, because we felt like there were lots of options on the market that required you to say, figure out, okay, what is the instance I need?

12:38How many instances do I need? What type of it? Okay, now I have to shard the data and blah, blah, blah. And coming from that data background, what I had been working with a lot is SQLite or DuckDB that just runs as part of your application code and would just talk to files that live anywhere. And it was super easy to install and use. So that's sort of what gave us that inspiration to make an embedded vector database. You had just got into this sort of idea of embeddings, or sorry, embedded databases, which, well, embeddings are related, but that's another topic. But the idea that LanceDB is embedded, you mentioned DuckDB and other things that kind of operate in the same sort of sphere.

13:25I'm wondering for those that maybe are trying to position LanceDB's vector database tooling, within a kind of wider ecosystem of vector databases and like plugins to other databases that support vector search. Could you explain a little bit about like, what does it mean that LanceDB is embedded? Like, what does that mean practically for the user? Maybe people aren't familiar with that term quite as much. So what does that mean practically for the user? And are there other kind of like general ways that you would differentiate LanceDB's tooling and the database versus some other things out there.

14:06So I love sort of geeking out about these topics. So at the very bottom layer, in terms of technology, I think there's a couple of things that fundamentally sets LanceDB apart. One, as you mentioned, is the fact that it's embedded or runs in process. I think we are one of two that can run in process in Python. we're the only one in JavaScript that runs in process. Number two is the fact that we have a totally new storage layer through LAN's column or format. What this allows us to do is add data management features on top of the index. And then number three is the fact that the indices, the vector indices and others in LAN CB are disk-based rather than memory-based so that it allows us to separate compute and storage and allows us to scale up a lot better.

14:57So those are kind of the big value propositions that these technological choices bring to users of LAN-CV. So number one, ease of use. Number two, hyper-scalability. Number three, cost-effectiveness. And then number four, the ability to manage all of your data together and not just the vectors, but also, if you think about it, the metadata and also the raw assets, whether they're images, text, or videos. Could you kind of describe a typical use case of a developer doing this where you're kind of taking those features that are distinguishing LanceDB from other possibilities, other competition, but just talk about what that workflow looks like or if there is a major one or a couple and just kind of get it very grounded so somebody that's listening can kind of understand how they're going to do it from A to Z when they're integrating LanceDB into their workflow?

15:53So there's a couple of sort of prototypical workflows that we see from our users. I think at the smaller scale for LanceDB, you're installing it via like PIP or NPM or something like that. And in general, you get some input data that comes in as like a Pandas data frame or maybe a Polar's data frame. And then you interface with an embedding model. You can do that yourself, or you can actually configure the land cb table to say hey use um open ai embeddings or hey use uh these hugging face embeddings land cb can actually take care of all that so it's a pretty quick sort of data frame to land cb and then you can search it and then that comes out as you know data frames or python dicks or things like that that plugs into the rest of your workflow that are likely data frame or pedantic or pythonic base.

16:48So that's number one. And then a kind of number two is really these large scale use cases where some of our users have anywhere from like 100 million to multiple billions of vectors in one table. And that's a much bigger production deployment. And typically, what makes LanceDB stands out in that area is one, it's very easy for them to process the data using a distributed engine like spark and they can write concurrently and get that done really quickly i think we're one of the few that offers gpu acceleration in terms of indexing so even for those really large data sets you can index pretty quickly and then number three is because we're able to actually separate the compute and storage even at that large vector size, you don't really need that many query nodes.

17:41You can actually just have one or two fairly average and commodity query nodes that runs on your storage of choice, depending on what latency requirements you want, and then just have a very simple architecture. For these types of architectures, the query nodes are stateless and they don't need to talk to each other. So when you need to scale up or when a node drops out, it has to come back in. There's no sort of leader election. There's no coordination. It really lowers the complexity of that whole stack. So another great example of this kind of architecture and the benefits that it brings is Neon, the Neon database.

18:19So I think Nikita, who's the founder, recently had a good Twitter thread about the difference between Neon and other databases. And he called it, you know, shared data versus shared nothing architecture. And I think that's also what we kind of strive to deliver in LanceDB versus other vector databases. Yeah, I know one of the things that I really enjoyed in trying out a lot of things with LanceDB is I can pull up a collab notebook, right and try out like I can import Lance DB I can import like a subset of the kind of database that I'm going to be working or the data that I'm working with it all runs fine I don't have to like set up some client server type of scenario and then when people ask well how are you going to how are you going to push this out to a larger scale the appeal of just saying hey well we can just throw up this Lance DB database on S3 and then connect to it.

19:22That's a very appealing thing for people because also those storage layers are available everywhere from on-prem to cloud to whatever sort of scenarios you're working with. So it's very, very flexible for people. Could you explain a little bit, because this is something like I've been asked a couple times, but so this is my selfish question because I have you on the line. So you're helping me with my own day-to-day work. But when I'm talking to some people, clients that I'm working with, I'm like, oh, we can just throw this up on S3 and then access it. Usually their question is something like, well, because they have in their mind a database, has a compute node, and somehow the performance of queries into the database is tied to the sizing of that compute node and maybe like how that's sort of clustered or sharded across the database.

20:18And then like this idea, oh, I'm just going to have even just a Lambda function that connects to S3 and does a query, right? Like this kind of, in some ways it like breaks things in people's mind. And so a lot of times their question was like, how does that work? How can a query to this large amount of data be efficient when the data is just like sitting there and in S3 or in another place? So could you help with help me with my answer, I guess, is what I'm asking? Yeah, absolutely. So this goes back to what we talked about earlier with separation of compute and storage. And if you've been sort of steeped in like data warehousing, data engineering land, this has been a big arc of data warehouse innovation in the past decade.

21:01by allowing us to scale up the storage versus the compute separately. This is the thing that makes the system seem magical, where you can process huge amounts of data on what seems like pretty commodity or pretty weak compute. And so the analogy that I like to make with these situations is kind of like, a lot of us are familiar with, let's say, like DuckDB demos or videos. And you can see instances where DuckDB is processing hundreds of gigabytes of data on just a laptop in a very vast amount of time. And they are able to spit out results almost interactively. And there are companies from MotherDuck to a new company called ValPlan that is looking to essentially distribute DuckDB queries on AWS Lambdas.

21:56It's basically the same thing. It's all about the separation of compute and storage. And that's only possible if you have the right underlying data architecture for storing vectors and the data itself. And just for someone that is not a database developer, can you describe in any words the generalities of that data structure that enables such a thing? Yeah, so it's two things. One is the columnar format. So typically, you know, from JNI to machine learning, you can have very wide tables, but typically a single query only needs like a couple of columns. So columnar format allows you to only have to fetch and look at like a very small subset of that data.

22:41Number two is that columnar format needs to have be paired with an index, like the vector index in this particular scenario. And that vector index, in order to give this separation of compute and storage, has to be based on disk. So you have to store the data on disk, not force the user to hold everything into memory, and then be able to access that very quickly. And then number three is how to connect that index with a columnar format. So a columnar format like Parquet does not give you the ability to do fast random access. So even if you had that good index using Parquet, you would not be able to get interactive performance in terms of queries.

23:25And it's only by having a new column or format like LANDS that can give you fast random access and fast scans that you can successfully put these two together and deliver the thing. So those are the three sort of big pillars that I think in our data architecture that makes this possible. While we were talking here, I'm going through GitHub on your repo and stuff, and was surprised at something that kind of prompting the next question. It looks like you're really addressing a wide range of different types of needs. And so there's obviously Python, as you would expect, but you have JavaScript. and then I was delighted to discover that there's a Rust client in there, which is when I'm not doing AI-specific things most of the time.

24:09That's my language of choice these days. Could you talk a little bit about kind of two things, the broader, like what you're trying to achieve, like how you choose what languages to support and how you're getting there, and then if you'll scratch my itch, what is your intention with that Rust client? Is it ready? What does it do? Just because I'm fascinated with that. Sorry. Yeah, absolutely. I love talking about Rust. The Rust package is actually not a client, but so on the core of both the data format and the vector database is actually in Rust. So the Rust crate that we have is actually the database or the embedded database.

24:46And so, and we actually build, for example, the JavaScript, again, the same thing with JavaScript. It's not just a client, but it's also an embedded database in JavaScript. So that is actually based on top of the Rust crate and kind of like you have in, say, like Polars or something like that. You have like a Rust core, and then you connect that into JavaScript. So we had actually started out in 2022 writing in C++. because Parquet is written in C++, you know, like serious data people and database people write in C++, right? Until they find Rust, of course. Right. And it was sort of a hack project during Christmas time in 2022, at the end of 2022, where we had to get a hack project for a customer actually, and where we had to actually re-implement partially the repath for Lance Format.

25:44And what we found was just It was so good that we decided to just actually rewrite everything in Rust. I think biggest things were we were a lot more productive. We rewrote roughly six months of solid C++ development in about three weeks with Rust. And this was like us learning Rust as beginners as we went along. A lot of that initial Rust code has again been rewritten over the past year. but it just made us feel a lot more productive. And then number two is the safety that Rust offers you has been amazing. With C++, like every release, it just didn't have a good feeling. It was almost like, you know, where's that next set fault going to come from?

26:32Whereas with Rust, you know, we felt very confident making multiple releases per week with, you know, major features. And we did not see anywhere near the sort of issues that we saw with C++. So everything has been really great. I know that Rust has become really popular now for, actually even with vector databases, like Quadrant, I think is Rust. Pinecone, they're not open source, but they publicly said that they've written their whole stack in Rust as well. One more question from you along the same line before I let it go, because we've hit that sweet spot that I love. Do you think, and this is not specific to Lance DB, but based on what you're saying, clearly you're thinking ahead on these things.

27:20As we go forward and you see both the AI applications and you see the different types of workflows and infrastructures, you know, becoming broader and more supportive, the multi-language aspect of getting out of only Python, for instance. Do you foresee that as a convergence where you're seeing language agnosticism developing in this space as it has in other areas of computer science? Or do you think that we're still going to be kind of locked in on the current sets of infrastructure and tooling, very Python-oriented, for the indefinite future? What is your thinking along those lines? So I think generative AI definitely changes the picture in that I think there's a very large TypeScript, JavaScript community that has been brought into the arena to build AI tools.

28:10And so I think this is also an underserved segment where it's not just vector databases, but data tooling in general lags far behind in JavaScript's life, TypeScript land versus Python. And I think there's a real opportunity for the open source community to create good tools for this part of the community as well. I want to hear about some of the actual use cases that you've seen people implement with LanceDB. Maybe if there's ones that stand out like, oh, this was cool because whatever it was, they used it at scale or it's like fits a very typical generative AI use case or whatever. And then maybe something that surprised you in terms of, oh, I didn't always when you put a project out into the world, there's these things where, oh, I really didn't expect people to be using it that way.

29:05But yeah, that sort of makes sense. So do you think of anything that fits into one or both of those categories? The use cases for LAN CB in the community that I see falls into three or four large buckets. One is, of course, generative AI, RAG, and things like that. And I think it's not so much the use of LAN CB that I think is really cool, but it's the applications that people build with it that is really cool and amazing. And I think a lot of the applications that people build that is cool, that really takes advantage of LAN CB is things where you need RAG to be very agile and that you need it to be really sort of tightly bundled with your application.

29:55You can sort of call this RAG from anywhere and have it return pretty quickly and without too much complexity. And so this is where I see a lot of folks from like your standard, like chat bots and chat with documentation to things like productivity tools where, you know, they build things that help people organize their daily schedules to, you know, much more high stakes things in production and like, you know, code generation or, you know, like healthcare and legal and things like that. And so there, I think typically you see vector data set sizes from like the tens of thousands up to single digit millions of vectors typically.

30:41And so production means you really scale up both the number of data sets that you have and then the number of vectors that you have. and one of the cool things that I've seen that takes advantage of LAN-CV and LAN format uniquely is there's a code analysis tool that sort of analyzes your GitHub repository and plugs it into a RAG, like customer success sort of tool. And what they want to be able to do is say, query the state of the database, like this today versus yesterday versus a week ago to say, hey, was this issue fixed or not? And like, what's still outstanding. And so LansCB uniquely gives you this ability to version your table and also do time travel.