Long Live Context Engineering - with Jeff Huber of Chroma

26:23And that actually brute forcing information curation is going to become extremely, extremely popular. Now today, the prospect of running 300 parallel LLM calls, even if it's not very expensive, you know, the tail latency on any one of those like 300 LLM calls, API availability, like it's also really bad. And so like there are good reasons to not do that today in a production application, but those will also go away over time. So those patterns I think I've seen emerge that are, that is a new thing that I think I've only seen start to really become popular in the last few months. and by popular I mean popular and the leading tip of the spear but I think we'll become a very dominant paradigm We've also covered a little bit especially on the code indexing side of the house so everything we've been talking about applies to all kinds of contexts I think code is obviously a special kind of context and corpus that you want to index We've had a couple of episodes the cloud code guys and the client guys talk about they don't embed or they don't index your code base they just give tools and use the tools of code search and I've often thought about whether or not this should be the primary context retrieval paradigm, where when you build an agent, you effectively call out to another agent with all these sort of recursive re-rankers and summarizers, or another agent with tools.

27:39Or do you sort of glom them onto a single agent? I don't know if you have an opinion, obviously, because agent is very ill-defined, but I'll just put it out there. You can pull that apart. So, you know, indexing by definition is a trade-off. Like when you index data, you're trading write-time performance, for query time performance. You're making it slower to ingest data, but much faster to query data, which obviously scales as data sets get larger. And so if you're only grepping very small, 15 file code bases, you probably don't have to index it, and that's okay. If you want to search all of the open source dependencies of that project, you all have done this before in VS Code or Cursor, you run a search over the Node Modules folder.

28:23It takes a really long time to run that search. that's a lot of data like to make that indexed and sort of again make that trade-off of write time performance or create time performance like that's what that's what indexing is like just like demystify it what is this right like that's what it is you know embeddings are known for semantic similarity today embeddings is just a generic concept of like information compression there's actually like many tools you can use embeddings for i think embeddings for code are still extremely early and underrated but regex is obviously an incredibly valuable tool and you know We've actually worked on now inside of Chroma, both single-loaded and distributed, we support regex search natively.

29:01So you can do regex search inside of Chroma because we've seen that as a very powerful tool for code search. It's great. And we built indexes to make regex search go fast at large data volumes. On the coding use case that you mentioned, another use case that, another feature we added to Chroma is the ability to do forking. So you can take an existing index and you can create a copy of that index in under 100 milliseconds for pennies. And in so doing, you then can just apply the diff for what file has changed to the new index. So any, like, corpus of data that's logically changing. So very fast re-indexing is the result.

29:36But now you can, like, have an index for, like, different each commit. So if you want to search different commits, search different branches or different release tags, like any corpus of data that's, like, logically versioned, you now can search all those versions very easily and very cheaply and cost-effectively. And so, yeah, I think that, you know, that's kind of how I sort of think about, like, regex and indexing. and embeddings. I mean, yeah, the needle continues to move here. I think that anybody who claims to have the answer, you just shouldn't listen to them. When you say that code embeddings are underrated, what do you think that is?

30:06Most people just take generic embedding models that are trained on the internet and they try to use them for code. And it works okay for some use cases, but does it work great for all use cases? I don't know. Another way to think about these different primitives and what they're useful for, fundamentally we're trying to find signal. Text search works really well. Lexical search, text search works really well. When the person who's writing the query knows the data. If I want to search my Google drive, I just, for the spreadsheet that has all my investors, I'm just going to type in cap table because I know there's a spreadsheet in my Google drive called cap table.

30:42Full text search. Great. It's perfect. I'm a subject matter expert in my data. Now, if you wanted to find that file and you didn't know that I had a spreadsheet called the cap table, you're going to type in the spreadsheet that has the list of all the investors. And of course, in embedding space and semantic space, that's going to match. And so I think, again, these are just like different tools and it depends on like who's writing the queries. It depends on what expertise they have in the data, like what blend of those tools is going to be the right fit. My guess is that like for code today, it's something like 90 % of queries or 85 % of queries can be satisfactorily run with regex.

31:17Regex is obviously like the dominant pattern used by Google Code Search, GitHub Code Search, but you maybe can get like 15 % or 10 % or 5 % improvement by also using embeddings. Very sophisticated teams also use embeddings for code as part of their code retrieval code search stack. And, you know, you shouldn't assume they just enjoy spending money on things unnecessarily. They're getting some, they're eking out some benefit there. And of course, like for companies that want to be like top of their game and want to like, you know, corner their market and want to serve their users the best, this is kind of what it means to build great software with AI.

31:5080 % is quite easy, but getting from 80 % to 100 % is where all the work is. And each point of improvement is a point on the board and is a point that I think users care about and is a point that you can use to, yeah, fundamentally just serve your users better. Do you have any thoughts on the developer experience versus agent experience? This is another case where, well, we should maybe reformat and rewrite the code in a way that it's easier to embed and then train models there. Where are you on that spectrum? Yeah, I mean, one tool that I've seen work well for some use cases is instead of just embedding the code, you first have an LLM generate a natural language description of what this code is doing.

32:31And either you embed just the natural language description, or you embed that and the code, or you embed them separately, and you put them into separate vector search indexes. Chunk rewriting is kind of like the broad category for what that is. again this is like the idea here is like it's related to indexing which is as much structured information as you can put into your write or your ingestion pipeline you should so all the metadata you can extract do it at ingestion all of the chunk writing you can do do it at ingestion if you really invest in like trying to extract as much signal and kind of pre-bake a bunch of the signals at the ingestion side i think it makes the downstream query task like much easier But also, you know, just because we're here, like it's worth saying, like people should be creating small golden data sets of what queries they want to work and what chunks should return.

33:24And then like they can quantitatively evaluate what matters. Maybe you don't need to do a lot of fancy stuff for your application. It's entirely possible that, again, just using regex or just using vector search, depending on the use case, that's maybe all you need. I guess, again, anybody who's claiming to know the answer, you should, the first thing you should ask is, let me see your data. And then if they don't have any data, then you have your answer already. I'll give a plug to the talk that you gave at the conference, how to look at your data. Yes, looking at your data is important, having going on data sets.

33:54These are all like good practices that I feel like somebody should put into like a little pamphlet, call it the Ten Commandments of AI Engineering or something. Okay, you might do that. Thou shalt look at your data. we're about to move on to memory but like i want us to sort of leave space for like you know any other threads that you feel like you always want to get on a soapbox about that yeah that's dangerous that's a dangerous thing to ask um i have one to to key off of because i think uh i didn't i didn't know i didn't know where to insert this in the conversation but we're kind of skirting near it that i'm trying to explore which is you know i think you had this rant about r a and g where the original transformer was sort of like an encoder decoder architecture.

34:37Then GBT turns most transformers into decoder only. But then we're also encoding with all the embedding models as encoder only models. So in some sense, we sort of decoupled the transformer into, first we encode everything with the encoder only model, put it into a vector database like Chroma. And Chroma also does other stuff, but you know. Then we decode with the LLMs. And I just think it's a very interesting meta learning about the overall architecture. It is stepping out of just the model to models and system. And I'm curious if you have any reflections on that or if you have any modifications to what I just said.

35:20I think there's some intuition there, which is the way we do things today is very crude and we'll feel very caveman in five or ten years. Why aren't we just, why are we going back to natural language? Why aren't we just passing the embeddings directly to the models who are just going to functionally re-put it into latent space, right? Yeah, they have a very thin embedding layer, yeah. Yeah, so I think there's a few things that I think might be true about retrieval systems in the future. So number one, they just stay in latent space. They don't go back to natural language. Number two, instead of doing this is actually starting out to change, which is really exciting, but for the longest time, we've done one retrieval per generation.

35:54You retrieve, and then you stream out a number of tokens. Why are we not continually retrieving as we need to. Don't call it that. Stop. But there was a paper in maybe like a GitHub that came out a few weeks ago. I think it was called unfortunately Ragar1 where they teach DeepCigar1, kind of give it the tool of how to retrieve. And so in its internal chain of thought and its infinite times compute, it's actually like searching. There's also retrieval augmented language models. I think this is an older paper. Yeah, there's a bunch of realm and retro and it's kind of a long history here. somehow not that popular I don't know why a lot of those have the problem where either the retriever or the language model has to be frozen and then the corpus can't change which most developers don't want to deal with the developer experience around I would say we would do it if the gains were that high or the labs don't want you to do it I don't know about because the labs have a huge amount of influence the labs have a huge amount of influence.

36:59I think it's also just like you don't get points on the board by doing that well. You're just like, no one cares. The status games don't reward you for solving their problem. So yeah, so broadly continual retrieval I think will be interesting to see come out of the scene, number one. Number two, staying in a betting space will be very interesting. And then, yeah, there's some interesting stuff also about kind of like GPUs and how you're kind of like paging information into memory on GPUs that I think can be done like much more efficiently. And this is more like five or ten years in the future that we're kind of thinking about but yeah i think i think when we look back and think this was like like hilariously crude the way we do things today maybe maybe not you know we're solving imo uh challenges with just language you know yeah it's great i'm still working on the implications of that like it's it's still a huge achievement but also very different than how i thought we would do things like you said that memory is the benefit of context engineering i think there's you had a random twitter about stop making memory for your eyes are complicated how do you think about memory and it's what are like maybe the other benefits of context engineering that maybe we were not connecting together i think memory is a good term it is very legible to a wide population again this is sort of just continuing the anthropomorphization of lms you know we ourselves understand how we are we as humans use memory we're very good at well some of us are very good at using memory to learn how to do tasks and then those learnings being like flexible to new environments and you know the idea of being able to like take an ai sit down next to an ai and then instruct it for 10 minutes or a few hours and kind of just like tell it what you want it to do and it does something and you say actually do this next time the same that you would with a human at the end of that 10 minutes at the end of those few hours the ai is able to do it now and the same level of reliability that a human could do it, like, is an incredibly attractive and exciting vision.

38:54I think that that will happen. And I think that memory, again, is like the memory is the term that, like, everybody can understand. Like, we all understand, our moms all understand, and the benefits of memory are also very appealing and very attractive. But what is memory under the hood? It's still just context engineering, I think, which is the domain of how do you put the right information into the context window. And so, yeah, I think of memory as the benefit. Context engineering is the tool that gives you that benefit. And there may be stuff as well. I mean, maybe there's some version of memory where it's like, oh, you're actually like using RL to improve the model through data scene.

39:31And so I'm not suggesting that like only change in the context is the only tool which, you know, gives you great performance on tasks. But I think it's a very important part. Do you see a big difference between synthesizing the memory, which is like based on this conversation, what is the implicit preference? Yeah, that's one side. And then there's the other side, which is based on this prompt, what are the memories that I should put in? I think they will be all fed by the same data. So the same feedback signals that tell you how to retrieve better will also tell you what to remember better. So I don't think they're actually different problems.

40:11I think they're the same problem. To me, the thing I'm wrestling with a little more is just um what are the structures of memory that makes sense so there's like obviously all these analogies with like long-term memory short-term memory let us trying to coin something around sleep i do think that there there definitely should be some sort of batch collection cycle maybe sort of garbage collection cycle where it's like where the lm is sleeping but i don't know what makes sense like we're making all these analogies based on what we think how we humans work yeah but maybe ai doesn't work the same way yeah i'm curious about uh anything that you've seen that's working yeah i always again you know as a through line of this conversation i always get a little bit nervous when we start creating new concepts and new acronyms for things and all of a sudden there's you know info charts that are like here are the 10 types of memory and you're like why these are actually if you squint they're all the same thing like do they have to have to be different you know like you have to blow the people's minds no i don't think you do I don't know.

41:09You got to resist the slot machine. The slot and the sloth machine. Compaction has always been a useful concept in... Even in databases. In databases on your computer. We all remember running defrag on our Windows machines in a 1998. And, you know, so yeah, again... Some of us not old enough to do that. I am. Not at this table. And yeah, so obviously offline processing is helpful. And I think that is also helpful in this case. And as we were talking about before, like what is the goal of indexing? The goal of indexing is to like trade write-time performance for query-time performance. Compaction is another tool in the toolbox of like sort of write-time performance.

41:51You're re-indexing data. It's not indexing, but actually it is indexing. It's sort of re-indexing, yeah. You're taking data, you're like, oh, maybe those two data points should be merged. Maybe they should be split. Maybe they should be like rewritten. Maybe there's new metadata we can extract from those. Like let's look at the signal of how our application is performing. Let's try to figure out like, Like, are we remembering the right things or not? Like, the idea that there's going to be, like, a lot of offline compute and inference under the hood that helps make AI systems continuously self-improve is a sure bet.

42:19Part of the sleep time compute thing that we talked about was pre-computing answers. So, based on the data that you have, what are likely questions that the person is going to ask? And then can you pre-compute those things? How do you think about that in terms of Chroma? We released a technical report maybe three months ago. The title is Generative Benchmarking. And the idea there is like, well, having a golden data set is really powerful. Having a, what a golden data set is, is you have a list of queries and you have a list of chunks of those queries should result in. And now you can say, okay, this retrieval strategy gives me, for these queries, gives me 80 % of those chunks.

42:56Whereas if I change the embedding model, now I get 90 % of those chunks. That is better. And then you also need to consider cost and speed and API reliability and other factors, obviously, when making good engineering decisions. But like you can measure now like changes to your system. And so what we noticed was like developers had the data, they had the chunks, they had the answers, but they didn't have the queries. We did a whole technical report around how do you teach an LLM to write good queries from chunks? Because you want like chunk query pairs. And so if you have the chunks, you need the queries.

43:29okay, we can have a human do some manual annotation, obviously, but humans are inconsistent and lazy and QA is hard. And so can we teach an LLM how to do that? And so we sort of did a whole technical report and proved a strategy for doing that well. So I think generating QA pairs is really important for benchmarking a retrieval system, golden dataset. Frankly, it's also the same dataset that you would use to fine-tune in many cases. And so, yeah, there's definitely something very underrated there. Yeah, I'll throw a plus one on that. I think as much attention as the context rock paper is getting, I feel like generative benchmarking was a bigger aha moment for me, just because I actually never came across a concept before.

44:07And I think like actually more people will apply it to their own personal situations. Whereas context fraud is just generally like, yeah, don't trust the models that much, but there's not much you can do about it except do better context engineering. Yeah. Yeah. Yes. Yes. Whereas generative benchmarking, you're like, yeah, generate your, your, your evals. And, and, you know, part of that is going, you're going to need the data sets And it'll sort of fall you into the place of all the better best practices that everyone advocates for. So, yeah, it's a very nice piece of work. I think having worked in applied machine learning developer tools now for 10 years, like the returns to a very high quality, small labeled data set are so high.

44:45Everybody thinks you have to have like a million examples or whatever. No, actually, just like a couple hundred, even like high quality examples is extremely beneficial. and customers all the time I say, hey, what you should do is say to your team, Thursday night, we're all going to be in the conference room, we're ordering pizza, and we're just going to have a data labeling party for a few hours. That's all it takes to bootstrap this. Google does this, OpenAI does this, Anthropic does this. You are not above doing this. Great. Yeah, exactly. Yeah, look at your data. It's what matters. Maybe you should classify that as label your data, not look at.

45:19Because look at seems a bit too... I agree with that, yeah. There's some more... View only. Right. I agree with that. Yeah. Yeah. Read and write. Read and write. While you mentioned it, I should correct myself. It wasn't standard cognition. It was standard cyborg. My favorite fact about you is you're also a cyborg with your leg. True. If you see Jeff in person, you should ask him about it. Or maybe not. Maybe don't. I don't know. I don't care. Don't care. Standard cyborg, Mighty Hive, and Know It. What were those lessons there that you're applying to Chroma? Yeah, more than I can count. I mean, it's a bit of a cliche.

45:50It's very hard to be self-reflective and honest with yourself. about a lot of this stuff, but I think viewing your life as being very short and kind of a, you know, a vapor in the wind and therefore like only doing the work that you absolutely love doing and only doing work that you doing that work with people that you love spending time with and serving customers that you love serving is a very useful like North star. Um, and you know, it may not be the North star to like print a ton of money in some sense. There may be faster ways to scam people into making$5 million or whatever. But if I reflect on, and I'm happy to go into more detail, obviously, but if I reflect on my prior experiences, I was always making trade-offs.

46:34I was making trade-offs with the people that I was working with, or I was making trade-offs with the customer that I was serving, or I was making trade-offs with the technology and how proud I was of it. And maybe it's sort of like an age thing, I don't know, but the older that I get, I just more and more want to do the best work that I can. And I want that work to not just be great work, but I also want it to be seen by the most number of people because ultimately that is what impact looks like. You know, impact is not inventing something great and then nobody using it. Like impact is inventing something great and as many people using it as possible.

47:01Is any of that, you know, and we can skip this question if it's sensitive, but like, is any of that guided by religion, by Christianity? And I only ask this because I think you're one of a growing number of openly, outwardly, positively religious people in the Valley. And I think that it's kind of what I want to explore. You know, I'm not like that religious myself, But I just kind of like, how does that inform how you view your impact, your choices? There was a little bit of that in what you just said, but I wanted to sort of tease that out more. I think increasingly modern society is nihilist.

47:38Nothing matters. It's absurdist, right? Everything's a farce. Everything is power. Everything's a comedy. Everything's a comedy. A meme, yeah. Yeah, exactly. And so, like, it's very rare, and I'm not saying that I always am the living exemplar of this, but, like, it's very rare to meet people that have genuine conviction about what flourishing for humanity looks like. And that's very rare to meet people that are, like, actually willing to sacrifice a lot to, like, make that happen. And to start things that, like, they may not actually see complete in their lifetimes. Like, it used to be commonplace that people would start projects that would take centuries to complete.

48:19Yeah. And, you know, now that's like less and less the case. The image that comes to mind is the Sagrada Familia in Barcelona, which I think was started like 300 years ago and is completing next year. Yeah. I've seen it in construction, but I can't wait to see it completed as well. Yeah. I'm sure the places are booked out already. Yeah. Yeah. And so, you know, it's common. There are actually, you know, a lot of like religions in Silicon Valley. I think AGI is also a religion. It has a problem of evil. We don't have enough intelligence. it has a solution a deus ex machina it has the second coming of christ that agi the singularity is going to come it's going to save humanity because we will now have infinite and free intelligence therefore all of our problems will be solved and uh you know we will live in sort of like the palm of grace for all eternity it's going to solve death right and so like i think that like religion still exists in silicon valley i think that it's like you know there's a conservation of religion you kind of can't get rid of it yeah um but the god gene yeah i mean you know people have different terms for this but uh like i think that i'm always skeptical of religions that haven't been around for more than five years put it that way yeah there's a survivorship bias anyway i i do think like you're one of the more prominent ones that i that i know of and uh i think you you guys are a force for good and i like to encourage more of that i don't know that you know people should believe in something bigger than themselves and yeah build for uh plant trees under which they will not sit.

49:47Am I mangling the quote? Is that actually a biblical quote? I don't think it's a biblical quote, but I like that quote. That's a good one. So yeah, plus one. I think society is really collapsed when you just live for yourself. That really is true. Agreed. Who does your design? Because all of your swag is great. Your office looks great. The website looks great. The docs look great. How much of that is your input? How much of that do you have somebody who just gets it? And how important is that to like making the brand part of the culture? I think all about, you know, again, going back to the Conway's law thing, like you ship your work chart, you ship what you care about as a founder in some sense.

50:25And like, I do care deeply about this aspect of what we do. And so I think it does, it does, it does come from me in some sense. I can't take at all credit for everything we've done. We've had the opportunity to work with some like really talented designers and we're hiring as well for that. So if people, you know, are listening to this and want to apply, please do. I think, I mean, it's cliche to decrib Patrick Collison quotes, but he does seem to be one of the, like, most sort of public embodiers of this idea that, like, how you do—and I'm not sure this is a dark quote from him, to be clear, this is more of just a broad aphorism, but, like, how you do one thing is how you do everything.

51:01and just ensuring that there's a consistent experience of what we're doing, where, like you said, if you come to our office, like it feels intentional and thoughtful. If you go to our website, it feels intentional and thoughtful. If you use our API, it feels intentional and thoughtful. If you go through an interview process, it feels intentional and purposeful. I think that's so easy to lose. It's just so easy to like lose that. And in some ways, the only way that you keep that is by insisting on that that standard remain. And I think that that is one of the main things that I can do really for the company as a leader.

51:41It's sort of cringe to say that you do kind of have to be the curator of taste. It's not that I have to stamp everything that goes out the door before it does. but at a minimum companies, you know, maybe it's not even like downhill and quality. It's not sort of legible that any one thing is bad or worse, but it's more like people just have their own expressions of what good looks like. And like, you know, they turn that up to 11 and then like the brand becomes incoherent. Like, what does this thing mean? And like, what do they stand for? Again, there's no longer a single voice. Yeah. I don't, again, I'm not claiming that I'm good, perfect at this or good at this, but we certainly wake up every day and we try.

52:19You have a lot of, it's very powerful, the skill you have to convey straightforward principles and values and thoughtfulness, I think, in everything that you do. You know I've been impressed with your work for a while. Thank you. Anything we're missing? You're hiring designers. Any other roles that you have open that you want people to apply for? If you're a great product designer that wants to work on developer tools, I think we have one of the most kind of unique opportunities at Chroma. If you are interested in extending the kind of research that we do, that's also an interesting opportunity.

52:55We're always also hiring very talented engineers that want to work with other people that are very passionate about kind of low-level distributed systems and in some ways solving all the hard problems so that application developers don't have to. When you say that, can you double-click on low-level distributed systems? people always say this and then like okay rust like like you know like linux kernel what are we talking here yeah i mean like that maybe like you know a useful encapsulation of this is like if you care deeply about things like rust or deterministic simulation testing or raft paxos tla plus consensus tla plus really um wow you know if you just keep them saying these are these These are like proxies for, you would like, you would like the work that we do here.

53:41I just really want to tease out the hiring message, but also I, um, part of my goal is also to try to identify who are, what is the type of engineer that people, that startups are really trying to hire and they cannot get because the better we can identify this thing, I can, you know, maybe create like some kind of branding around it, create an event and like get these, like there's a, there's a supply side and a demand side and they can't find each other. And that's why I put AI engineer together was that, that was, that was part of it. But then this distributed systems person, which like I have heard from you and like a hundred other startups, what is the skill set?

54:15What are they called? What do they do? And part of that is like, part of that is cloud engineering, because a lot of times you're just dealing with AWS. Sure. A lot of that, a lot of times you're just dealing with, I don't know, debugging network calls and consistency things if you're doing replication or whatever. Where do they go? What do they do? Yeah. Yeah. Like, but they don't use TLA plus at work, you know? Probably not. Yeah. I mean, last year I started the SF Systems group. Yes, the reading group. Yeah, there's presentations. And the point of that was like, let's create a meeting place for people that care about this topic because there wasn't really a place in the Bay Area for people to do that.

54:54So that continues to go now and continues to run, which is great. I mean, to be clear, we have a lot of people on the team who are extremely good at this. And so it's not that we have zero, it's that we have six or seven. And you won 20. but yeah it's not that we want more but we are in some ways like i feel like our product roadmap is very obvious and we know exactly what we need to build for the next even like 18 months but quality is always a limiting function quality and focus are always limiting functions and like well yes i will always make my land acknowledgement to mythical man month eventually like it's good more people you do you kind of do need more people because you need more focus like you need more people to care deeply about the work that they do.

55:36And the AI is certainly an accelerant and it's helpful. And it's a reason that our team is still very small today relative to many of our competitors is because I think we've really embraced those tools. Your cursor shop, Cloud Code, Windsurf? People use whatever they want. So I think all those tools get some usage internally. So far we've still not found that really any AI coding tools are particularly good at Rust though. I'm not sure why that is. Other than the obvious, there's just that many examples of great rust on the internet. And so, um, you know, yeah. You would think that, you know, rust errors would be, help you debug itself, right?

56:16You would think. Apparently not. Okay. All right. I have zero experience in that front. Yeah. Uh, I've, I've contributed three things to the rest SDK of Temporal and that was my total experience with rust, but I think it's definitely on the rise. It's, it's zig, it's rust. and I don't know if there's a third cool languages. I think ghost accounts. Golang, yeah. If you're in that bucket, reach out to Jeff. Otherwise, I think we're good. Thanks for coming on. Thanks for having me, guys. Good to see you. Thank you.