Chroma and Agentic Retrieval

27:22So it's a 20 billion parameter model. And when we trained it, initially, the accuracy of pruning, I think, was around 80%. So 80 % of the time, it selected the right documents to remove. So that basically makes the prune chunks tool for the base model kind of a non-starter. You're going to have an immediate 20 % accuracy loss. By the end, I think we were around 95 plus accurate on a wide variety of tasks after trading. And I think on our most recent checkpoints internally, it's like closer to 97, 98 % accuracy on the prune tool. And most of the loss in accuracy comes from the search itself. And the model itself is quite good at discerning what's accurate and relevant versus not.

28:00So that ability to self edit the context ends up leading to a very large reduction in the overall token usage. And what that does is it also makes it so that you can do inference a lot faster. Because we're keeping the context window relatively bounded at any given period of time, around 32 ,000 tokens, the pre-fill phase, which is the first phase of language model inference, ends up being relatively fast. And then the decode phase is relatively fast as well, because we're keeping it fairly bounded at all times. So we can afford to pay this repeated pre-fill cost because we're pruning the context and the model is small enough that we don't kind of feel that penalty from that trade-off.

28:38And there's obviously some subtlety there in like, how much should you prune versus not? When should you prune versus not? And we're also starting to work on, can you actually like use latency as a training objective and let the model actually self-optimize around latency at train time? And so that it kind of learns the most optimal pruning strategy, not from an accuracy perspective, but also from a latency perspective. So there's all these subtle tricks you have to do in order to make it work but fundamentally i think the core contributing idea on the harness side of the model the thing the set of tools it has and the way that the model behaves is this idea of self-editing its own context and self-editing its own history i think if we just like look at how like what the result of this is i think it was stated that your context one at 20b matches this isn't that long ago so opus 4.5 sonnet 4.5 gemini 3.1 pro on sort of most of your benchmarks that seems pretty impressive it has an understatement so i mean like what's kind of your read on why a purpose trained small model can actually get there i guess i think the core belief is if you start to decompose what a model does when it searches the set of behaviors it has are more valuable than the reasoning that it's doing So there's a program that you can follow or an algorithm that you can follow to do search.

29:58And a lot of the time that algorithm is a set of very narrow tasks chained together. And if you can just figure out the right time to deploy those tasks, and if you can figure out how to be good at those subtasks, then you can be good at agentic search. Fundamentally, I think the set of tasks is how do I look at a high level user query and then decompose it into a plan of lower level queries to issue against the search system. How do I look at a page and determine its relevance? How do I look at a page and determine what follow-up queries to make? And then how do I analyze the set of results I have at the end and determine what could be relevant, right?

30:34Which is a different relevance determination. So I think that set of tasks, which are fairly narrow, you can optimize for in isolation through a set of training examples. And so actually in training context one, we have a staged curriculum where early on in the lifecycle of training, we only give it problems that are easier and need one to two hops to satisfy. And then later on, we start to like expand the difficulty of the tasks that it has. And then we also do two phases of training. It's not only trained through reinforcement learning, we also have an additional supervised fine tuning stage, which uses synthetic data and data distilled from open weights models.

31:09So we take open weights models, and we distill their behaviors into the smaller model. And that initial set of behaviors alone actually shows you a very large performance increase. But the thing you're after is grounding the model in the right set of priors around how should I behave when I'm in certain cases. And once you get that into the model, it's pretty easy to optimize once you have those basic set of behaviors. Even just getting the model to tool call in parallel gives it a huge advantage in its ability to very quickly reduce the number of turns that it needs in order to discover information.

31:40And GPT-OSS 20B doesn't natively do parallel tool calls. And so if you just, you know, do some amount of SFT or fine tuning on the model in order to teach it that behavior from the data, it very quickly gets better at that and leverages it even before you do any sort of reinforcement learning. So I think the reason that you can get these small models to be good at these tasks is because they don't require a lot of reasoning capacity, and they don't require a lot of subtle decision making. And we have done some work to show that for ourselves, we've done some work to show that, oftentimes having world knowledge does help you.

32:13Like if you know information about a given topic and you've memorized information about a given topic, you're going to be able to issue better queries. And so a larger language model will probably always outperform a smaller one at steady state, but the smaller ones are cheap enough to fine tune using LoRa that on narrow enough corpuses, you can very quickly also fine tune a small model on a narrow corpus and make it better at individualized corpuses and personalize the model for specific tasks. And I think that's where the advantage really starts to shine is can you actually just give it a map of the corpus or teach it the vocabulary of a corpus so it's able to search it better?

32:49The same way like if you and I go and search our Google drives, we already know what's in there. So we're going to be able to search for it more effectively. You can see that sort of behavior in smaller models as well. Yeah, that's really interesting. I think just to wrap up context one, the bit that might not be clear or surprising even to some people is that you actually release the weights open source like under Apache license so what was the thinking there I mean on SE Daily we're always very pro open source that's fantastic but every company has probably gone through a thought process of why to do that yeah so Chroma is an open source company like even our core database is Apache 2.0 I think there's many reasons to do open source before Chroma I wouldn't say I was the largest you know open source zealot but I've over the years you know become a card-carrying open source member or whatever you want to call it, where I'm quite zealous about things being open source.

33:41So I think there is like some amount of just personal belief and philosophy inside the company that things should be open source where they make sense. And I think it's fundamentally like, one, we wanted to learn very quickly, and more people are going to use the model if it's open source and tell us what's working and what's not working. More people are going to be interested in building with it, more people are going to be interested in integrating it into other places where we may not be able to integrate it. And also the model is very small, so you can run it locally. And the way that Chroma initially grew as a product in the market was we offered a local version of the database.

34:13And so we're quite fond of solutions that meet developers where they are, which is on their laptops or on their machines at home and let them tinker around and experiment with technologies that are close to the frontier in a way that doesn't cost them an arm and a leg in order to play around with it. And maybe it won't be the case that every subsequent model we release is open source. But at least for the time being, I don't see a reason for us not to continue to do things in an open source fashion. I think it's positive some. I think that we learn when other people release their work, we benefit greatly from it.

34:44And we obviously benefit from OpenWeights models, both quite literally in the training of Context 1, but then also just in our usage of them. And so I think mostly for us, it's just a matter of getting people to use it, learning very quickly, and then contributing back to an ecosystem. And our belief is that our ability to grow that ecosystem at this moment in time, AI is still very early, even though it feels like the world is almost over or whatever people on the news want to tell you. AI is still very early in its lifecycle. And I think that right now is not the time to be zero sum in our thinking with respect to it.

35:18Yeah, that's great. I mean, we've talked to quite a few companies who have decided to move from closed source to open source over the last year, for example. And I think we're seeing more and more of that. Either the company just starts day one that way. I work for a company that started that way as well. Or actually, it doesn't look great, actually, if you're working on anything on the frontier that is all closed source. And in Anthropics' case, you can be closed source and then somebody leaks the whole code anyway. So you might as well be working in the open. So definitely support that. If we just look at the product stack of Chroma, there's database, there's sync, and the agent product.

35:55Can you just kind of walk us through where does a developer start, maybe? or like it does not matter just the right tool for the job or is there some classic rqc of like how a developer starts and then adopts more of the product yeah i think most people their first interaction with chroma to date is the local embedded version of the database so you pip install a package and all of a sudden you're in a jupyter notebook or you're just coding in in typescript and something is running locally on your machine and then we have a cli that lets you once you're done prototyping, if you need to upload your data, because unlike SQL databases or like traditional, you know, the use cases that relational databases serve, it actually does make sense to have this notion of pushing data up from staging to production.

36:42With these kinds of applications, you might curate some sort of knowledge base or curate some data set and then push it up. So that's the place that most people start. And then they use our CLI to upload the data to our cloud product once they're ready to start going in production. There's two ways they can do that. Either they can use the database directly if they want to control their chunking strategy and their embedding model choice and, you know, choose all of these nuanced parameters. But over the years, I think things have kind of coalesced onto a set of best practices. You should use sparse vectors with splayed.

37:10You should use a dense vector embedding model that's of a sufficient quality, but not too large. Turns out you don't need 4096 dimensions. You're very much okay with a smaller dimensionality embedding model. I mean, you'll get just as much performance in terms of accuracy at a much lower cost footprint. If you want to just have those choices all made for you, you use our sync product and you just, you know, you can either dump your data in an S3 bucket or upload a file or connect it to one of our supported sources. So like GitHub, a website, and then, you know, we're soon releasing support for a bunch of other upstream connectors.

37:42And then we'll just automatically keep those sources in sync with your database, with your Chroma. And then that's kind of the normal arc. Context one then becomes something that people, you know, once they start to think about how do I actually really get high quality search in a low latency manner, they usually start adopting that part of the stack. I think agentic search is still new enough that people tend to be adopting it fairly late into their product development. But I guess, do you see that as maybe accelerating into the future? Sure. Yeah. Now people are coming to us, especially after we released it.

38:17Now people are coming to us and it's kind of the starting point for their product development. And that's been really interesting too, which is, you know, how can we, you actually start to build your product differently if you start with that frame of mind. If you start with like, no, an agent is going to do the querying, you start to actually build your product differently. So I've had people come to me and be like, you know, while the agent's doing the querying, why should humans do any of the editing? And so actually they have products now where like, you know, users actually don't edit information directly.

38:43they're only allowed to edit through the model because they think that the model will be able to do a superior job of traversing the information space and making all the right edits in the right place and keeping all the information in line. I think it's also changing how people think about their product development. This notion of like, you know, not everything needs to be a set of checkboxes and forms and sliders. Actually, you should just drive everything through natural language and push everything into the agent. Yeah, I definitely see the same thing where I sit. And it's definitely just an interesting time to be building products where every conversation is, is this a human or an agent?

39:15When you're talking about how it's being used, we're always saying, is it the user or is it an agent? And how does that differ? And I just find that super fun and interesting that that's kind of where we are now when it comes to building product. If we sort of look at how a larger enterprise would want to use something like this, you guys have quite a bit of material on what's called BYOC. So bring your own cloud. could you like talk to me a bit about when in chroma's journey did this come about and why and small anecdote from talking to someone who is in the current yc batch and they were actually asking me and saying oh we're being pushed to you know buy a certain big customer to do byoc and i said really think about that strongly because that's a big undertaking and of course they're going to say they want that, but don't take that on lightly.

40:03I think you could probably help expand on why not to take that on lightly. Yeah. I think Databricks was kind of famous for doing this like hybrid deployment model early on in their life cycle. And I think when they asked the CEO of Databricks, what are two things you would do differently? One of the two things was he was like, I wouldn't do the hybrid deployment model. I would have just skipped straight to the other way of doing it, which is maybe kind of privileged hindsight, right? Maybe you wouldn't have succeeded if you hadn't done the hybrid deployment model. So BYOC for us means we deploy our code into a customer's VPC, and then we get back some amount of observability into that system in order to operate it in management.

40:42And we keep our control plane on our side, and then we're able to kind of monitor the health of the system, perform operational procedures, updates, upgrades, scaling, some basic logging and telemetry comes to us, and then you get the experience of a managed service, but you don't have to deal with actually running the infrastructure yourself. It's a very easy setup. You just kind of, you know, apply some Terraform and run a Helm chart and you're up and running. And then our team kind of handles the day-to-day for you, all the day-to-day operations. And we actually had the idea to do BYOC from day one.

41:12I think if you don't choose to do it from day one, it's really, really, really difficult to bolt on because your entire architecture hinges on the fact that you're doing BYOC. And actually it was an immense pain to convince everyone on the team that this was worth doing, because it's really painful to build. It's really, really hard to build a system in this way. It makes you have to do things in a much more difficult way upfront, greatly sacrifices your velocity. You have to draw boundaries in your system architecture that feel very unnatural to draw at times. Or you could argue, I'd actually say they're boundaries you should have, but ones that you don't necessarily need right then and there.

41:47And I think that if you don't really need to do it, I wouldn't advise that you do it unless it's like quite core for your ability to grow your business. For us, it's really important because the customers that we want to serve that are at a large scale, which is where our product, the core database really becomes useful, is at a large scale. If you have a small volume of data, other solutions are fine. If you only have like 10 ,000 records, you probably don't need scale out horizontal serverless search engine. You're probably fine just throwing all your data in a Postgres database and it will work.

42:18There's no reason to use our cloud product at that point, like other than just ease of experience and, you know, just wanting something that really fits AI applications natively. But I think unless you really need to, I would not recommend it. And I think that my advice to this person in their YC batch would be like, make sure you're going to make a lot of money if you do that. Make sure that it's like something that you think you can repeat and deploy because it's a very large distraction. But I do think that if it makes sense to you, it is kind of an increasing trend. And And I've noticed that buyers especially are pushing for it, where buyers have started to kind of learn that this is a deployment model.

42:51It gets through security. It gets through compliance, ticks all the boxes. And that is probably reason enough to invest in doing it if it can result in a meaningful outcome for your company. That said, when BYOC started, I think it was pretty debated whether it was a deployment model that made sense at all. There was a lot of well-known people online who would tell you that, you know, this is the silliest thing ever. No one should do it. It actually makes absolutely no sense. There's blog posts you can go read about this. I think it's one of those things that's a little bit of an organizational tax, right?

43:23You have to make the compliance people happy. You have to make the security people happy. You want something that ultimately makes you able to sleep at night and give you control. And I think BYOC is great for that. Whether it's actually secure, more secure or less secure, I think that really ultimately depends. I think that at the end of the day, you still have to do your due diligence and you still have to check on the buyer side what's going on under the hood. I don't think it reduces that work or makes it go away. Yeah, I definitely agree with everything you've just said. On the control plane side, do you get any pushback that the control plane does still live with you?

43:55And how do you, I guess, talk to companies about that? Yeah, I think there's various degrees of pushback. When we initially started talking to customers about doing BYOC, there were some customers who were quite negative about having any data on our side, even telemetry. That's just a matter of education. The product can't work if we don't have basic telemetry on the system. So at this point, everything that we store on our side of the fence is just metadata. It's what data exists in the system, what is the utilization of the system, what is the version and epoch counters of various pieces of information in the system.

44:35Nothing is directly revealing private information. And because it's open source, we literally just send people the code. We're like, here, here's the control plane code. You can read the schemas. You can look at it yourself and you can make your own determinations as to what we're saying. You don't have to trust us entirely. So I think that ability to just literally in conversation, you know, just send someone the code and just be like, here, this is exactly what I'm talking about. You can poke through it and read it. And now even with agents, you know, people will just quite literally, we have this like Slack bot agent and a customer will be asking us in Slack, hey, can you tell me like what data is stored?

45:08And we can just paste the code and then ask the agent, can you traverse all the files related to this and then compile a list of everything that is related and you get an extremely fresh and accurate grounded answer about what data we capture and it just points to the code. I think that for us has been how we've alleviated people's concerns about what we're capturing and storing. And then the core of it is that we have the idea that no ports are open on the data plane. So everything is done through kind of a reverse tunnel approach where the data plane reaches out to us. We don't ever go in and reach into the data plane.

45:38And that's also part of it is if you can just come and say like, hey, it's your VPC by default, no ports are open, everything comes on a push based model out. So even telemetry, we kind of buffer and then allow it to push out the telemetry. But then also like operations, we don't reach into the data plane in order to do operations. There's kind of a service that runs inside of the data plane that we post a message and then it pulls checks. Is there any operation that needs to be done that gets pulled down into the data plane? And then that operation gets performed. And that's kind of like tax, right?

46:08Ideally, you wouldn't build it that way. That's a very strange way to build some sort of operations thing where you're like pulling for operations and then, you know, applying the operation. So that's a lot of moving parts in order to do a very simple thing inside your cluster, which ideally you would just be able to issue a post request and boom, the operation is completed. Now you have to do it in a slightly different way. although because we're built on kubernetes you can kind of leverage a lot of the hooks that kubernetes gives you to do declarative progression in the same way and benefit from that but it's still you know requires a lot more work than you might otherwise want to take on yeah i think that's a really good summary and as you say i think starting on kubernetes is a huge help as an infrastructure product as yourselves it kind of seems like the only way but there will be other products out there that don't start that way and does make something like a byoc deployment even more complex yeah if you came to me six months in and you were like we're doing by seo i'd have been like we can't we absolutely can't it's too hard to bolt on yeah no that's very interesting so i'll probably go back to this yc founder and give him some tips on this basis as well so just looking i guess ahead the future obviously postgres is having a other moment these days and i think people just say oh well pg vector is like good enough for things like this like where do you you see things going as like, are we, I guess, asking the devil advocate here of like, could things converge back to, you know, a frontier large model is actually going to get good enough at this stuff and context isn't such a thing anymore?

47:32Or like, how are you guys thinking about it? And like, how are you thinking of building into the future? I think there will always be a need for managing the context of language models. I think the problem of what should I know right now for a language model is a problem that will always exist. In language models today, if you just went to your chat GPT and you were like, tell me my schedule for the day, even if you hook in the MCPs and stuff, a lot of the time they won't call it the right tool. They won't say the right thing at the right time. And so I think there will always be a need for, can you get the right information into the model?

48:05And that's fundamentally the problem that we're interested in solving. I think about it a lot like the CPU versus the file system. The language model is the CPU. And then there's a disk off to the side. And there's a ton of complexity inside of your hard disk and the interface to that hard disk and the controller and the operating system that talks to it. And then the firmware on the hard disk. I think that that interface is the one that our company believes in, which is there is a language model. It acts as this new frontier natural language information processing system. And then there is a subsystem off to the side that decides what information that model should see and decides what information should be stored and decides what information needs to be potentially updated and consolidated with existing information.

48:51That's the system that we in the long term want to build. Today it looks a lot like a database but then where we're going if you can kind of tell with how we're building components on top of the database right why is a database company training its own models is because our belief is that soon it won't be I have a database and I have intelligence. It'll be intelligence actually runs inside of the database. And a lot of the way that the database actually performs its function is by leveraging intelligence subsystems. And you need to think about those two things at the same time for performance reasons and also just for domain awareness.

49:24It is very cost prohibitive to move large volumes of data around and put them into representations for various downstream processing systems and do all of this conversion and ETL and then do all of this reformatting and re-indexing and re-parsing. And so if you can build a system that manages that all in one place and makes it very cheap to move compute closer to the data by leveraging object storage, by leveraging a shared storage layer, such that if I want to, for example, just run asynchronous language model jobs on my data, I can. If I want to co-locate the node that literally has the query indices with the GPU, such that now my GPU can talk not to a remote service, but talk to a local language model, I can.

50:07Because this year, I think it'll be very common for us to, as consumers, use language models that run at 1 ,000 tokens per second. Last year, we were all running with language models that ran at 100 tokens per second. And so it's not inconceivable that in a couple of years, we're going to have language models that run at 10 ,000 tokens per second. And so today, all of these search systems make the assumption that a network call and the latency to go to your search system is fine because, hey, the language model is slow. But I don't think the language model will always be slow. And putting these two systems next to each other such that you can put intelligence and data in a low latency path next to each other, and maybe even in the future bypass the CPU entirely.

50:46Why does the language model need to traverse indices via the CPU? Can it just directly have a subkernel launch on the GPU that does the search of the index in line in a very low latency fashion, and then feed that data directly back into the language model. I think those sorts of systems will exist. And our goal long term is to be in a place in the market that we can build towards that. I think that that's kind of what we've been building towards all these years is you should be able to put there's some system that will need to exist that you can put any data into any unstructured data, and then it can get any data and then it can satisfy any query on the way out.

51:22And I don't think that system will look like a Postgres. I think that system will look like something much different. And I think it'll look vastly different than what we have today. But if we can get the right abstractions in place and get the right building blocks in place, maybe we'll get an opportunity to earn the right to build that system in the future. Yeah, amazing. Well, on a sort of total sense, I think that's a great place to leave it. Two more questions. The first is a usual of just like up and running for a developer. But I feel like in this case, is it a website? or should we be saying what CLI command do you start with or you know what would be best yeah pip install chroma db chroma db run is all you need to do to get up and running but then if you're interested in learning more try chroma.com great and yeah just a curious one for me what's in the name chroma where did that come from that is a great question i think it's just one of those things where it's it's short it's easy to say it's memorable and then it has a nice ring to it yeah you guys have a nice aesthetic as well around the whole product as well so definitely do encourage people to go to chroma.com and check that out because i think you guys have done a great job with that as well so amazing to have you on so much detail really learned a lot myself today i'm sure the audience did as well so really appreciate it and we'll be following along with with chroma thank you so much for having me

52:45Thank you.