Turbopuffer with Simon Hørup Eskildsen

23:49I would love for there to be more of a playground. What we see our customers do is generally export the namespace locally and then try to visualize it. I don't know how many of our customers do visualize the data and how many of them just start writing evals against it. I would love to help with tooling, but we've been very focused on the database itself. Okay. Yeah, no, I think it's really helpful to understand. It probably also speaks to where, obviously, where Togo Puffer has been focusing, maybe perhaps versus some other products that sort of are trying to like hit all the points at once and maybe not doing a bit of a general approach shall we say let's go to performance and i believe that the term in vector stores is recall so that effectively trade-off between latency and accuracy i believe turbo puffer kind of does measurement on i believe at least samples one percent of all queries internally and you have presented back that data anonymized through blog posts as well but talk to us about that?

24:45Like, how do you measure? Why are you measuring? And where do we go from there? Recall is extremely important, because if you're building a pipeline that searches, you don't want to think about in your evals, whether your search engine is inaccurate, you just don't want to think about that, that should be the job of the vendor, or this database that you've chosen, you should not have to manually tune it, and you should not have to manually run recall against it. Turbo Puffer, of course, as we reiterate on our ANN algorithm, we run it against, of course, various benchmarks that we have internally, right?

25:16Academic benchmarks and so on to make sure that it performs. But nothing beats real world performance, right? In a real world, someone is going to insert the same song a million times into a cluster with lots of other songs. They're going to put that burgundy skirt in there like 20 million times, right? And not realize they have the bug and they still are going to expect that the accuracy is going to go up. So overall, accuracy is very important to us, and we consider it our job. I want to just briefly explain what recall is, and then I'll talk a bit about how we work on it at TurboCoffer. So recall, if we go back to my original example, right, of the only way to get the absolute true result of the top 10 closest vector embeddings to another query and vector embedding is to look at the entire dataset, right, O of N.

26:01With the A and N, what you do is, if you issue a query with the approximates near and neighbor index, then you compare the top, let's say top 10 for the approximate results and the exact results. If you have a recall of 90%, it means that nine of the 10 results were correct in the ANN result. Of course, if it's one or 100%, then everything is overlapping. We find that our customers are very happy somewhere between 90 and 100 % erring on around 95 % recall, right? So it averaged across everything. That's what recall means. And generally anything above 90 % is really good. Recall is a tricky metric because you could search for banana in a cluster or like in a data set that only knows about songs.

26:46And it's going to give you a result because in some way, you know, banana is close to something, right? And so the vector distance might be very long, but it also in the clustered index, the longer you are away, the further the clusters are away from the query vector, the worse the results get. So it's a flawed metric, but it's like the best flawed metric that we've got and that we're all measuring against. So we're aiming for 90 to 100 % recall, erring on the side of 95. And when we, very early on in TurboPuffer's history, we decided to really take this on an hour problem. And as you mentioned, a percentage of production queries are sampled in production.

27:19So it means that for a random percentage of like somewhere around 1%, but it sort of scales how many queries a user does. We will send a query over to a different worker fleet and it will evaluate that like against the exact result, the approximate result and compare it and then report the number back to our instrumentation in Datadog. And so we have a dashboard, right? Where we look at every single customer and look at their recall and we will get the big red dots if someone's recall is below 90%. And that's how we reevaluate it because I think production is the only thing that tells the true story.

27:53You can't just evaluate against academic benchmarks. I think being on call for 10 years almost at Shopify taught me that nothing matters other than production. And it's going to be the same for accuracy, is that the only way that we will trust our recall is if we know that on all the production data sets, it's above 90%. Got it. I believe there is something that comes into this also, and native filtering. Perhaps you could talk to us a little bit about that. Filtering is the most important thing with recall, because this is where it gets tricky. And this is where just tapping on a vector index to an existing database is not quite enough to ensure that a recall is high.

28:30So let's take some simple examples and work through them, right? And explain what this pre-filter, post-filter, in-place filtering, whatever means, right? Let's say that you have a query that is, we'll keep using an e-commerce example here, and we're using it on a very large e-commerce data set. And we want to filter out any products that are not public. Let's say that 99 % of all the products are public, right? Like 1 % is sort of like held back as people are iterating on them or haven't released them yet or they've sold out. If only 1 % of the data doesn't match the filter, it's probably fine to just over filter by around 1 % and then filter it out after.

29:08That's a post filter. And with that, if you evaluate the recall, again, the only accurate way is to evaluate it against everything. You will get very, very high recall with a post filter. But you could imagine in a case where you're matching against an example that would only match, let's say, 10 % of the data set, it could be everything that, I don't know, everything that is the color blue, and that's only 10 % of the data set. Well, in that case, if that's 10 % of the data set, and I've overfetched 100 items, well, I'm not going to get that 100 % recall, right? The math just doesn't really work out for the precision that you need.

29:43So in that case, well, it's only 10 % of the data set. So maybe what we can do is we can just find everything and read another index, like a traditional B2E index or a bitmap index that's blue, and then evaluate all the vectors. And that works okay for something that's like 10%. It will be a little bit slow because that's a lot of vectors to look at, but it might be okay. But the trickiest queries are the ones in between, right? They're the ones that filter out maybe 50 % of the data sets. Imagine an example like you're in Singapore. Everything that ships to Singapore, maybe that's 50 % of the catalog, right?

30:13So it's like, okay, do you post-filter? you have to overfetch by a lot. You have to get a lot of stuff to make sure that you have the right recall because it might in a clustered index, it might completely eliminate some clusters. Like maybe, you know, food items are never shipping to Singapore, but you're searching for a banana and it's just not matching that much of it. But it might go into the next like food color clothing that might start shipping, right? But those clusters that you've cut off from. And when you start to think about it in the cluster sense, the query planner really has to be aware of like how much of different clusters match the filter and how much does that cut off and then you have to have some heuristics around how many vectors you have to look at in the order from the query vector to look at enough vectors that you can guarantee a high recall i realize this is hard to parse but really what i'm trying to impress here is that the query planner that plans how much data the query needs to look at to get high recall needs to be very aware of both the vector index and also the filtered index to get high recall and most databases that have just slap the vector index onto an existing database will only pre or post filter.

31:14Often the user has to choose. So you have to know about the selectivity of their data set, but it's not a trivial thing to choose. Yeah. And that leads quite nicely into what I wanted to touch on next, which is vector indexes. I believe it's SPFresh vector indexes that's used at TurboPuffer. This is probably a concept at a very high level that most of our audience are at least familiar with to some degree. If you've worked with databases, you probably understand the concept of an index, i.e. some way of saying to the database, hey, these are the kinds of queries we're going to be making on a very regular basis.

31:48So we need an index across, say, these three columns on this table, because that's the kind of lookup that we're going to be doing. That's a relational database index, for example. How does that look, obviously, in the vector context? And again, the choice here, SPFresh, I'd love to hear about that. So in the graph-based indexes, we talked about how there's graph-based and cluster-based indexes. In the graph-based indexes, they're really nice because you can add things to it and it fits neatly into the graph and you can just keep adding to them. And you don't have to worry that much about recall because the recall on a graph-based index is usually phenomenal.

32:23That's why they've been so popular because you just add to them. But given that we chose a cluster-based index for the read reasons that I mentioned before, when you do a clustered index, it's suddenly, you know, someone could have started adding products in a category or a cluster that didn't really exist when you created the initial clusters. And generally, a clustered algorithm sort of has to look at all the data and then decide what the clusters are. But when you get into millions or hundreds of millions or even billions of vectors, that can take easily days. Like on a state-of-the-art algorithm, it can take days on a very large machine to figure out what the clusters are.

32:56So you start using GPUs to try to do it faster, but it becomes very, very difficult to do. So basically these clusters can't be incrementally maintained, right? In a B tree, in a traditional database index, it's really nice because you just sort of like insert them into the tree and it nicely balances. And there's lots of properties that make sure you can just incrementally add to it like a graph. So in a clustered index to maintain it incrementally, you need a bunch of heuristics to maintain the clusters. You can think about it as like maybe you know if we continue with the e-commerce example you have a shoe cluster and this customer is just like they just keep adding shoes and at some point the shoe cluster has maybe a thousand items in it and it's like okay well that's a lot of data to search every time we search for something relevant to shoes so we have to split the cluster so at some point you know every time you write it's like well this this is a shoe this is a shoe and then it reaches some terminal size of the cluster and we split the cluster so you split the cluster and then maybe there's a sneaker cluster and there is a leather shoe cluster and that's what it's decided to make the two clusters and you can imagine incrementally maintaining the clusters like this where they reach some size we split them once in a while you remove enough items that we have to merge clusters and every time we do this we have to recompute the centroids of the clusters it is much more complicated than that but that is the general idea behind spfresh that with enough of these heuristics you can do this this is probably the most complicated part of the entire terrible buffer code base to make this work at very, very large scale and make it work with recall and make it work with filters.

34:24That is how SPFresh works. And this scales very, very well. Building agentic AI apps isn't just about choosing the best LLM. Agents need short-term memory, long-term recall, and lightning-fast retrieval. Without it, you're left with clunky prototypes that never scale. You know Redis, the world's fastest caching solution? It turns out fast data is the key to good context. and good context is essential for fast, accurate memory. It's what makes AI agents actually work with your data. Redis for AI. The right infrastructure, the right tools, the only way to scale. Learn more at redis.io slash genai.

35:03Building an app often feels like a balancing act. You want to ship features fast. Chat, activity feeds, moderation, video. But building them from scratch is slow and complex. That's where Stream comes in. Stream provides developer-friendly APIs that let you add real-time communication without reinventing the wheel. Why Stream? First, developer experience. Stream has clean, open-source SDKs and great docs, and you can get a proof-of-concept running in hours. Second, speed to build. Experiment, prototype, or launch features quickly. No credit card required to start. Stream also scales. Over 2 ,000 global apps, including Strava, Patreon, Nextdoor, Robinhood, and Peloton, rely on Stream to power in-app communication for more than a billion users.

35:55Whether you're a startup or an enterprise, Stream handles the hard parts so you can focus on what makes your app unique. Get started today at getstream.io slash podcast. and moving kind of to scaling generally so my experience with just this space i know that the people approaching it are doing incredibly different ways is in the email space and for us namespacing was kind of a like a hard requirement so let's take that example for a second you know we've been talking about e-commerce which i'm also quite familiar with but let's hard life into email for a second so email you know you've got however many thousands of users and you absolutely never want someone to be searching and someone else's email turns up in that search that would be sort of catastrophic so namespacing i.e the idea that every user's data is in a very siloed sort of environment was incredibly important to us i'm aware that namespacing is a part of turbo buffer and scaling that has i believe it's like 14 million namespaces or something so talk to us about that because i mean oh sorry 40 million 40 million namespaces and i believe that's probably Probably also something that, again, maybe your Shopify experience in terms of that was a huge scaling challenge, I imagine, led into kind of how you've been able to approach this as well.

37:13Yeah, look, the only way you can shard anything or you can scale anything is to shard it. And so we decided at TurboPuffer to make namespacing a core sharding primitive that we exposed to the user. Because if you can give us your sharding key in a simple way, well, then we can scale really, really far with you. In TurboPuffer, a namespace matched directly to one shard. and a namespace is just a prefix on obik storage so you can imagine that if you have gregor dash email well that's prefix number one right and it's literally just like you know that on s3 gregor dash email slash right and then all your files are in there then you have simon dash email slash and so on so on so on and that way we are only constrained in our scalability on how many namespaces S3 can have?

37:59Well, S3 can have a lot of namespaces. We have yet to see any limit and there's no documented limit on how big this can be. And TurboPuffer, yeah, has more than 100 million of these namespaces. And this works great because it also means that we can encrypt every individual namespace differently because you might decide that you want to encrypt the Gregor email namespace or prefix on S3 with a key that you have access to. And I want to do that with mine as well to add even another layer of defense that's basically as good of a protection as if you had that data in your own bucket. There's no logical difference.

38:31You can rotate the key at any time or revoke it. So that namespacing is a core tenant of Turbo Puffer. And it's a core tenant of, you know, like I'm thinking back to my Shopify experience, right? And most of the problems we solved by abusing the wicked shard on the shop and that shops did naturally have ways that they to talk to other shops other than in some edge cases so that's exactly what we did at turbo puffer and build that in as a core primitive in the future probably a namespace will map to multiple shards and things like that but it will be abstracted against this like single layer of completely horizontally scalable search nice yeah as i say there was as you got a core primitive that i was looking for surprisingly let's just say other people in the space take slightly odd approaches to it or virtually no approach at all so i think it's something that you know the audience should look out for when thinking about this kind of thing so we're going to take a bit of a turn into who's actually using this because there's some huge names using turbo puffer notion and cursor are probably i imagine two of the biggest names if you like but also two of the names i'm i think most of our audience will be very familiar with and can probably start to imagine sort of how from everything we've just been talking about how the technology actually works under the hood like how that's being translated through to what does that even mean for them as a user each day?

39:46So maybe could we talk about both of those cases? I think super fascinating to hear about those. Yeah, we can start with Cursor. So I got to know the Cursor team in 2023. And when they saw the first announcement of TurboPuffer, knowing the team so well now, it's very clear that they just had a conversation around the dinner table at some point thinking, well, why hasn't anyone built it like this, right? They're big fans of S3, just like I am. So it slotted right into their mental model. And we went back and forth with a bunch of bullet point lists on email and spend some time with the team. And we were just completely aligned on what needed to be built here.

40:19There was a couple of missing features. So we built that out for them and then they migrated. For them, the storage architecture of TurboPuffer just made sense, right? When you open a code base in Cursor, it indexes the code base, right? And one of the parts of the indexing is to embed the code base. And this powers a lot of the agents and a lot of the functionality inside of Cursor then has the ability to do a semantic search on the code base. I use this all the time to be like, where does it do this? And where does it do that? And I can just do that in plain language. So Cursor, yeah, will embed with their own embedding models that they've trained, will embed the entire code base and then use that as a tool call into everything else.

40:55And for them, the storage architecture of having everything in memory, which was the previous solution that they were on, just didn't make a lot of sense. You don't need every code base ever opening Cursor in memory at all times. That's incredibly expensive. and the per-user economics of that just weren't sustainable for them. So they needed a way to bring the per-user cost to something that made sense to them. And so the TurboPuffer model of inflate the puffer fish when you're querying the code base just made a lot of sense, right? Only some percentage of the code bases are going to be at active at once, but they're still valuable to keep around.

41:26So it's just such a clear fit to the storage architecture. So they've been great. And their first bill was reduced by 95 % to move on to TurboPuffer. And they've been amazing partners that have inspired a lot of how we built TurboPuffer since then. Yeah, Notion is another customer of ours. And they also, it's a very similar story where they were using a vector database that also had things in memory. And the per-user economics, again, just didn't make a lot of sense to them. Again, only a percentage of these workspaces are active at once. So that really appealed to them about TurboPuffer as they wanted to connect a lot of this data into their LLMs.

42:01So it's a very similar story to the Cursor One. But I don't want to give the impression that TurboPuffer is only good for these use cases. I mean, it's still fundamentally the cheapest way to store data in the cloud is to put it in object storage at$0.02 per gigabyte and then cache it on a compute node only when it's actually being used. Because generally, in a traditional storage architecture, you have to replicate the data to three nodes, right? So in TurboPuffer's case, you have it in$0.02 a gigabyte of object storage, and then you store it on disk and memory on a blended cost of maybe around$0.10 per gigabyte.

42:32but on a traditional storage architecture with memory and disk and all of that you easily run into dollars per gigabyte so this allows us to have this pricing that just makes a lot of sense and it allows us to have a truly serverless environment where you can just like push in the vectors and this was great for both notion and cursor because they have a lot of rights right every time you edit files every time you edit in notion it has to issue a right to update the vectors in turbo buffer yeah that's interesting i have noticed just how fast cursor is indexing you know i love using cursor workspaces so you know like dumping say like three different code bases into a workspace and then having it kind of figure things out across all three and yeah the indexing has always impressed me so now i know why i mean obviously credit to cursor as well but it's great to understand the technology behind the scenes and yeah i was at a notion event not so long ago and obviously they're doing a big push on their ai offering and a lot of this is look we're going to index across all these data types you know like google drive and i believe gmail is coming soon and slack and well probably slack i mean there was that announcement about the api restrictions but i'm sure maybe they're figuring that one out but at the end of the day it's a lot of data that needs to be indexed and then effectively ai rag search over that so that's also super interesting that turbo popper is kind of behind the scenes on that one yeah i think every company has a lot of data that they want to make it into context.

43:53And if you have a lot of data, then the economics of adopting another data store rather than perhaps using a vector index in your relational database could start to make sense when you get into the tens of millions of documents that need to be indexed, right? Same reason why we've always moved search workloads out after some point in time for full-text search as well. After a certain threshold, the downsides of having another data store start to make sense. So when you want to connect a lot of data to AI, we think the storage architecture makes a lot of sense. You touched on it, just talking through the cursor and notion examples pricing.

44:23I think it's good to call out Turbo Puffer doesn't offer any kind of free plan. And I think you've been quite sort of vocal about why you don't offer one. I think it'd be great to just to kind of hear the why's behind that. I fully support that just to be clear. But I think it's also great for developers to sort of understand as much for their own products as it is for the fact that they might want to come now try Turbo Puffer and then might be disappointed that there isn't a free plan, for example. Yeah, I mean, first of all, if you're not having a good time in the first 30 days and you've tried the product earnestly, then you have the right to cancel.

44:51But we think that the companies that benefit the most from Turbo Puffer, this is not a price point that's scary. And for small companies, it might make sense to use a small index that's tacked onto the existing relational database. But if you have ambition to index tens of millions of vectors, it can make sense to start on Turbo Puffer. But in a lot of cases, it's really about wanting to provide people a really good experience. And it's easier to provide people a really good experience when you have a commercial only offering and we can put behind the necessary staff to support people when they have questions and give them a really good experience so that you get the feeling that we're part of your team absolutely and you've announced i believe quite recently you're now ga so general availability how does turbo puffer sort of as an org now look i haven't done research and i always find it interesting are you two people are you 200 i mean where is turbo puffer yeah i mean we're less than 20 people and it's a very engineering heavy team.

45:43We've hired a bunch of database engineers, and that's the majority of the team, people working on the database. And now we're starting to hire other types of roles to expand the team and especially support our customers and so on. Yeah, that's the size of the company. We really want to have a dense P99 engineering environment. And so we try to hold the standards high on a team that we put in front of our customers and are developing this product. right this is a very nice number team-wise you know i think it's been a little bit over reported on sort of this idea of the you know the one person unicorn and all this nonsense so i think it's good to sort of call out that some of the the best companies coming out at the moment are maybe 10 but 20 also sounds good and a bit more and like from what you can share where's turbo puffer kind of going you know over the next say six months to a year like what's kind of top of mind for where the product needs to go one of the biggest frontiers we're pushing right now is just more database features.

46:39So for example, we just launched like conditional writes. So it's the ability to say, Hey, I only want to update this document if it's actually newer than the document that's in the database. So it's database features. It's a lot on the full-text search side. So Turbo Puffer doesn't just do vector search. It also does full-text search. And we see more and more of our customers doing that. There's a lot of expectations about what you can do in full-text search. And so we have to build that on the storage engine that we built from the ground up, right, to work. So those are the biggest things that we have.

47:04We have a very solid generic LSM storage engine and underpinning all of this that we've matured and now it's really about features and it's about it's about even more performance than we deliver now awesome well we're coming up for time one final question i had was just actually completely nothing to do with databases specifically it's actually around i mean you've talked about the name but also the visual language and just i guess the front-facing site very fun it's sort of pixel art i guess if you want to kind of give it some kind of name where did kind of the idea come from and like who kind of leads that on the team yeah i think it's like neo retro pixel art or something i don't know what the aesthetic is called yeah really this came from when i was back home in denmark and we had some some friends visiting and i was talking to to my friend who's a designer about this idea of of turbo puffer and i think he said later that he had no idea what i was talking about but i sounded really excited about it and he was so encouraging and in my mind lateral to the exactly how to lay out the bytes on disk and build a storage engine and index was sort of this just this aesthetic that i wanted and so we were riffing back and forth on what that would look like and kind of creating some sample sites and it was just like just like very bare bones right like i've you know at shopify i spent 10 years evaluating databases so i was on the buy side the majority of my career and every time i went to a website it's just like what does it cost what are the trade-offs who are the customers and what they're using for is these are the only questions i really care about and most database websites are just so filled with so many different other things right and i'm just like no what are the trade-offs like any database has trade-offs like is this the right set of trade-offs for me what's the architecture what are the guarantees what's the consistency model and so this show don't tell just was very important to me and then we wanted to breathe a little bit of fun into it i think turbo puffer in itself is kind of a whimsical name and so this aesthetic just sort of appeared over many many iterations that you'll see on internet archive and we're just really fond of it and will continue to iterate on it because it makes us happy when we go to the website yeah as you've called out if you go on turbo puffer and you'll see a whole bunch of like sliders etc so you get sort of into the meat of it very quickly but i think as you call out having this kind of fun aesthetic around it definitely sets the tone for who you are as a company as well reminds me a little bit in different form of you're probably familiar with tiger beetle and you know financial database and so they have a very very specific type of database but they also bring fun visuals to it which just and obviously the name so it's very memorable to that point i remember seeing something about turbo puffer and hacker news a little while ago and then this episode was suggested i knew the name straight away could visualize the website before it even come back to it so i think it's very smart so yeah thanks so much for coming on where can people find you or like where's the best place to go to kind of just get acquainted with turbo puffer yeah turbo puffer.com is a great answer point we post on all the social medias you'll find the link there.

50:00I'm Syrupsin on X where, you know, these days I mostly tweet when I go out for runs for some reason, but that's where you'll find me and find the company. Awesome. Well, thank you so much, Simon. Very deep technical episode today, which I think a lot of the audience will love and just really look forward to following along. Turbo Puffer, I think you guys are doing very interesting things. So thanks for coming on. Thank you so much for inviting me on.

50:29Thank you.