Moving Beyond RAG with Precomputed Context

27:45In knowledge graphs, especially RDF, you're even separating that kind of schema layer from the fact layer. In the first step, by generating the knowledge graph schema, by identifying the interesting entities, you can also generate the constraints around that. In the RDF world, I don't think we want to get into a discussion around which representation is best, but there's a long debate, shackle, et cetera. So you have a lot of options there to express constraints, but I would imagine this is probably that way. Right now, I said, I think you started the easy part, you generate a schema, which is already a set of constraints, right?

28:24You can put on then constraints on that. Hey, you always know that, I wanted to say, I mean, revenue needs to be like greater or equal zero. I have to think about it now from a business context, but just to make up an example, like you can, for example, say certain numbers need to be like strictly positive because otherwise they don't make sense. And you know that because you have identified that entity, like number of attendees in a meeting, for example, cannot be negative to come up with a good example there. And others might be then related. And I think the more time you spend analyzing that data set and also the bigger it is, right?

29:04It might be something you actually discover over time when you get more and more data in. You discover like new constraints. Again, there's a lot of work on knowledge graphs trying to identify these constraints and separate that schema layer and constraint layer from the actual fact layer in Knowledge Graph. So I would imagine this is probably a way where you can get that next step of always adding more boundary conditions. And then if we're a conflicting data set, you discover them, you can actually call that out. And I think that's valuable information by itself. If you can flag, hey, there's a conflict in the data set, probably at the beginning, that will require human interjection.

29:51But if you're learning that like, hey, this one data set, it's lagging behind and that's why it's not reliable. You can actually start, for example, ignoring the last two days because you know that second data set you're joining with, it's not reliable for the last two days. And then you only generate it like two days after if you have, for example, daily data ingestion. So I think this is where this curation step can then learn over time from kind of constraint violations. Right now we're kind of like dreaming. So I don't, I'm not aware of like any system doing that as of right now, at least.

30:26I'd love to dream a little bit more on this because this is one I've been thinking about as well. And in particular, like in my day job, I do a lot of work with things that are like essentially information about people and relationships where there isn't necessarily a ground truth that's going to be correct. And so one of the things I've been pondering is like, Can you create some sort of like Bayesian approach or confidence interval where you're trying to like, including that in the metadata for each particular piece of context and then like looking for ways to up your confidence or maybe even asking the end user?

30:57I mean, if we stay in the knowledge graph field, right, you can just have edges with confidence, right? So you can basically have an attribute on the edge. To take your example, right, I imagine you would build kind of like a network of people and then they have some relationships, know each other to start with our following to take like a Twitter slash X example. I think there you probably would have a ground truth, but you can just add an attribute on each edge saying this is the confidence I have there. And again, this is something which might evolve over time. If you're getting new information, Bayesian, as you mentioned, you can update your beliefs about that edge being true or not true.

31:41So I think that's basically if you view the context assets knowledge graph, you can embed the probabilities as part of that. I think that's probably the clean approach I would imagine at least like in quick as of now thoughts. Yeah, well, and it gets to this interesting question that we sort of alluded to earlier of like, how do you design this for effective agent use, right? Like what is going to make the agent most able to take advantage of this? And it might be that you don't want to give it those shades of gray. You just want to like have a cutoff. If it's above this confidence, it's in. And if it's below this, it's out.

32:20Yeah. And I think this is where agents are a bit different than humans, right? If we look at how is data consumed, so also just from a general database systems perspective, we as humans or even like dashboards, we're going to send over a SQL query or any query. We expect like one result and then we're going to fly with this result and just go forward. I think agents are both on the one hand empowered to use more because they can iterate, right? So an agent, I might give that result back. I'm only 5 % confident that this is actually true. And depending on which scenario I'm in, this might still be valuable information for the agent.

33:08And I think this is then where I think we can view that from two perspectives. I think, first of all, if we come from a governance perspective, and I think this is something we are seeing a lot with like Nexus as well. I actually, I want like a constraint, like anything where I'm less than 50 % certain, don't even give out to the agent. From a generalist perspective, if I trust the agent, I just want to give it as much information as possible. even this like, hey, I'm only 5 % certain in that can be something useful for the agent to just decide, like, what is the next step? What is my next planning step?

33:44Hopefully, it's not going to use the information, but it can use that meta information to actually take a different decision for its next step going forward, rather than just getting like, oh, no information available around that, right? So I think this comes down a little bit to the different use kits, whether you want to expose that information. But I think with agents compared to human users, because they iterate so quickly and they can form like a new plan on a much, I don't like the term cheaper, but just kind of like faster level, right? Because as a human, I start thinking about it and then I'm in like a second latency range.

34:21But from like an agent perspective, if I'm getting the outcome, oh, it's only 5 % certain in that, I can then turn around my question and not ask for specific data, but I could ask like, what facts are you more certain than threshold, for example? So I can actually change my approach for ring about data. Well, that brings us to a question about like, how do we expose this to agents? And I'm kind of curious, I saw that with Nexus, you have like your own little dynamic no QL language or something along those domains. But yeah, I was thinking about like, to what extent is this a pre-query that's pre-loading some things in context if we know it versus this is just a tool exposed to the agent?

35:01Or how do you think about the right ways to give agents access to this context? So I think from an agent perspective, I need a combined answer of data plus metadata. So for example, that answer we just talked about earlier, like I believe the answer is five, but I'm only 10 % certain about it. First of all, if I can get both those facts back, This is something valuable. NoQL, for example, and also we were trying to evolve and also get like more standardization around that. This is, for example, the other thing we have seen, which is super valuable for agents. If I can expose schema, at least schema on the output I want, I want that result that should be US dollars.

35:53So if I'm asking, like, what is the price of X? And I, as an agent, I can specify in a deterministic format and really want to make sure that I'm getting it in U.S. dollars as opposed to euro, for example. And I can specify that as part of my structured query, right? So no QL is kind of that pair of structured query where I can also interject metadata. It's not just a SQL query. I can actually put more in there. And secondly, I'm getting the same on the response side. I'm not just getting like the structure is the table back as with a SQL table, but I'm actually getting data combined with that metadata plus certain constraints are automatically enforced.

36:40So we can automatically check that all of those are numbers because I said like that price field, it should be a positive number lower than 10 ,000, for example. Got it. I think we may just, just to, just to step out. I think in that, in that agent interaction, how do you think about it? Because in most cases, it's not going to be like a one shot interaction. I'm not like getting one response back. I am typically trying to give back and interact in this pairs of data and metadata, because that allows both sides to actually give more relevant answers and understand the context to reuse that term once more around the data which has been given, right?

37:25It's not like one shot interaction, one result going back, but it's actually the combination which makes it powerful. Of course, also on the downside of those loops, they can get expensive, right, if we just talk token usage. And I think maybe this is coming back to where we talked about earlier, like this curation. I think we want to keep that loop of planning as, we know we're like super deep on that right the less layers we can have in there we have to do on the fly the better so i think if there is pre-curated context you have to do that only once you have to do this loop of actually discovering you do that once up front that might be expensive but then in the next interaction you can actually benefit from it and we already know what to give you back and we don't have we have to do like one iteration pipe less because there's still going to be plenty of set going on in the system yeah i've heard it described as like llms are really good at things that they can sort of describe linearly and the more layers of abstraction you have to handle all at once the more they are both it's expensive it takes a lot of their own like internal expensive key value memory to to track that context going through but it's also less reliable and so when you can like flatten those steps so So for each LLM interaction, it's only having to track essentially a smaller linear interaction, but it may sub-call out to things that are doing those depth.

38:56I find your linear abstraction. I find that interesting. I'm usually like in my mind, I'm still seeing that as a loop, but I know what you mean with linear. I'm just trying to avoid like a sub-loop at like one step in that loop. Exactly. So if I can avoid that level of nesting of loops, the better. Yeah. Yeah, well, and you want to offload that to your context system, right? So it's doing that sub loop. It also has to deal with exactly one layer of loop. Exactly. I mean, sorry, it's just coming to my mind right now. If you look at the software engineering, right? It's kind of like it's an abstraction, right?

39:30In software engineering, I'm also trying to pull out like common functionality into like one function. Because first of all, for us, it makes it easier to maintain code. I can reason about like one sub part much better because I can also treat it independently. So I can treat this like one sub loop, which is focused on curation or preparing that like independent of that larger loop retrieving it. So it's kind of like composability and software engineering as well. Oh, absolutely. Well, and I think one of the like superpowers for dealing with LLMs and programming is thinking in terms of domain-specific languages because LLMs are very linguistic, right?

40:11They think in language. And so if you can create a language that's the shape of your problem, LLMs are very, very good at utilizing that. And then to your point, right? You decompose that. Okay, now I have a new set of abstractions I need to build. I need to build those different functions that create my DSL. Yeah, I mean, I'm always a bit careful because you can go overboard with that as well and then make it too complex. But I think to a degree, the less we as humans have to, I mean, this is the power of abstraction, right? We don't have to keep everything in mind, but we can only talk about an abstraction here.

40:45And I think the same is true for the LLM. I think you described it very nicely. Like if you don't have to go in the sub loop, which is also different contexts, right? So basically the LLM has to change its role potentially and still wants to store kind of its previous context, but it's doing actually a different role. the more we can avoid that the easier same as for us humans if i'm looking at a code and there's an abstraction for a class with certain rules i only have to kind of rationalize and think about that abstraction and then at another time i can actually go into the implementation of that if i want to but from like that high level code i can actually stay at that interface level so i think i think that's interesting coming back to what you're doing with nexus is you're essentially doing that same thing for data, right?

41:32You're creating a domain-specific data object, a thing that is not only abstracting out the code layer that generates it, but abstracting out kind of the form of data that's going to be useful for a particular problem domain. Yes. So I guess the thing I'd be wondering about there, and this is more of an observational thing if you've seen a bunch of, how much variation is there in what makes for that shape to be useful? Obviously the details of what's exposed, the permissions, those things are going to vary, but like how similar are the shapes across these different types of context datasets? I think the shape, especially if it comes to the entities we're talking about, this is different between different datasets, but I don't think the recipe changes.

42:18If you had to represent it as like a class hierarchy of what's playing together, I think that underlying structure is going to be very similar. So the scheme of it is going to be similar. the actual instantiation is going to be different. So what entities do I have? But usually I'm having some core entities. And I think we said earlier, if we promise to come back to like that narrow knowledge graph representation, we actually do have and we do detect relationships between entities as well. So I think this pattern of identifying a small, fairly small core subset of entities, which are central to that, right?

43:00It's going back to like document analysis, like different scores of identifying what's relevant to a document. It's still like very, very much here. I think we're using different algorithms, different ways to detect that. But I think overall, we're still trying to identify what is core to this document, what is core to this data set. And then from that, we can identify that small, potentially related subset of entities We parse it into that. And on the side, we're having the vector search for finding similar instances of these entities. We're having the kind of more structured representation, and then we're having the metadata next to it.

43:41So I think that it doesn't vary so much. It varies just from the different instance creations. I'm curious. So if I use the very particular domain I'm in, one additional thing we often find useful is having essentially quotes that support whatever the extracted entity is. I'm curious if that's a pattern that you see. Yeah. I mean, what we do is, I think we talked about in the very beginning, right, the lineage. So we always keep a trace back to where it's coming from in the original document. So kind of like a pointer back to the original document. because yes, you're 100 % right from like multiple angles.

44:18I think, first of all, in many cases, we actually would recommend the system to retrieve the ground truth in addition, right? So if it has a chance, if it's not too much content, it retrieves the ground truth as well because it might even give you more information. And I think this was already true in retrieval, right? I think a typical retrieval pattern is you find the similar vector, similar entity in the vector space, but then you still kind of retrieve the original data to back that up and feed in. And then I think this is still the case also with context. I think with context, because we already have a structured representation, I have to do that less.

44:56But we definitely, we give back a pointer back to the original data, to the original place in your document. And again, coming back to then also the version document, right? If my data set is evolving over time, we need to make sure we are referencing the right version of that. Yeah, that makes sense. I'd love to dig a little bit into this knowledge graph concept, and in particular, the fuzzy vector-based or semantic knowledge graph. So how are you thinking about those relationships in a world that maybe is not as perfectly structured as your classic knowledge graph? So I think the interesting aspect of this, I think that's something we didn't talk too much into deep about yet.

45:36I think what is interesting is that even though imagine the data set being static, right? I'm creating like one context artifact. I might create like one context graph, one knowledge graph instance associated with that. Over time, when I'm traversing that, I can actually keep modifying my knowledge graph. When I'm initially creating my context, I have a certain budget of compute. Imagine every night your GPU cluster lies idle, and you're actually going to show that on your previously curated context, even though the initial data hasn't changed. You can try to find, I like your example earlier, like imagine in your knowledge graph, you have certain probability values where you think this is true, that edge should be in the knowledge graph or not.

46:28And then overnight or whenever you've got free compute time or idle compute time, you actually keep curating your knowledge graph and either collapsing it, expanding it, using graph machine learning to identify new connections, try to group things into certain clusters, and actually keep curating it, potentially also a Bayesian approach with additional questions I've seen come in. So I actually get more data, not by the initial data changing, but by actually just seeing the questions coming in. So I know what I might want to answer. And that can help me curate that over time. And I know right now we talk a lot about knowledge graphs.

47:11I don't think that necessarily has to be a knowledge graph. I can do the same curation over time by spending more effort on it. If it's in different representations, I can do the same. if it's in a more relational format, potentially with additional edges in between those, with like semi-edges, right? I can always put a graph format into relational schema as well by encoding that as joins. Might not be efficient to retrieve, but I can do it. So I think even there it's true. So even if I don't have like a traditional knowledge graph, I can still maintain that over time and kind of keep curating my context artifact and making it better, especially if I've seen some of the questions being asked against it.

47:55So that's interesting, actually, because it leads me down a path of like, can you even change the schema of that context artifact? Because you see, for example, hey, we're reliably getting questions that end up having to query across two different sources of context or things like that. Maybe we should provide a combined view or what have you. Denormalization, right? And I remember that, well, it's like 20 years ago, teaching the different normal forms in relational database systems. Most certainly, I think this is so nice that you have the abstraction of a context, right? A context is telling you, hey, this is actually the schema I have in there.

48:34So from like a consumer perspective, I'm still, I'm not changing the interface, even though I'm changing the schema inside. I just have like version two, version three, version four of my context. but it can have a different internal representation. That's awesome. Maybe kind of like a container. I mean, I've worked on data products before. You could even see that it's like a small data product combining the data and metadata into one entity. And then as a data product or in the microservice world, like the Docker container, it gives you still like an interface. You can change the internal representation and evolve that over time as well.

49:14Yeah, very good question. Well, and it's interesting too, because like you have some sets of data and especially metadata where you're wanting to do repeatable deterministic types of things on it. And so you don't want that to change too much. But a lot of this is like prepared for an agent to consume as a programmer, you can treat it as a black box and change that, that internal representation. We talked about NoQL before, right? So kind of like our query interface there. I think this is one of the nice things there because in NoQL, I can basically also give you a schema I expect back, right?

49:48So I'm keeping that interface static. But again, because the interface is given by NoQL and we know how to map it, we can change the internal representation and might be more or less efficient to retrieve. But I have that flexibility through that abstraction. So all of this, we're kind of talking about a layer that is developing now that is on top of these graph databases like Pinecone originally was, or we talked about being on top of a relational database or something like that. Do you see this as being a separated abstract layer that there's going to be a set of different options out there in the world?

50:26Or is this something that's going to be tightly integrated and vertically stacked with these kind of underlying data stores? So I think for us, it's actually we benefit a lot from owning the entire stack. If I look at our implementation, at least, we benefit a lot from owning that end-to-end because we can just expose metadata. So freshness metadata for the underlying vector data is as we know how to get them. We don't have to duplicate that potentially getting out of sync. Maybe just to give one example. So I think for us, we actually benefit from implementing it really end-to-end. And I think this is giving us better performance.

51:11I think the other aspect where it's going to come in is governance. We talked about different access, permission levels, different service accounts, having access to different things. So in theory, yes, you can definitely develop an abstraction layer over that. I think that will take quite some time to identify what do we all need to expose there. So what I would imagine is probably at the beginning, different people will develop different solutions. And then certain standards are going to evolve. Like how we're thinking about it, I think for us, the layer, which probably is going to standardize first, is going to be a little bit on this query front, of course, the NoQL.

51:59And then from there, let's see where it takes us, right? We can also probably standardize some of the knowledge definition, the knowledge spec. But I said, like right now, also just from like a governance perspective, keeping track of lineage, it is a very beneficial of for us controlling the entire stack. I would at least give it like another two years of iteration. And then maybe we have identified all the patterns and we can drive that out into a general spec. Yeah, that makes sense, right? We're very much in this phase of the tech isn't actually good enough yet. And so there's a lot of benefits to squeeze out every piece from that vertical integration.

52:37and we'll get to a place where, oh, okay, now we've over-served that and it benefits us to split it apart and optimize different pieces. So I think it's going to start at the query front. With query, I mean actually query and response, right? So that kind of format. And then it's probably going to go down to that stack. It's just going to make, I think, some of the benefits you get from that, except like lineages, I think it's something super helpful to be able to trace it back to the initial days. That's going to get a bit tough if we go into a general abstraction to just get it through like these different systems.

53:14But who knows, lads? I'm looking forward to talking to you in a year again and see how we have evolved then. Yeah, absolutely. Well, we're coming close to the end of our time and we've covered a lot. But is there anything we haven't talked about that you think would be important to discuss before we wrap? No, I think we actually covered it very nicely, right? I think we looked a little bit at the history, what actually led us to develop something like Nexus. I think that problem like agents actually retrieve, how can we build a system for agent retrieval? How can we make that efficient? So I think that's something very nice where we started.

53:51I think we branched up into many different areas. So I think we really covered most of that. Also, we already covered what I had here in my last notes, kind of the outlook. I think the outlook where this is going to grow is we're going to see more and more use of integrations with different semantic layers. I said, like right now, we do that internally. We're going to see like a lot of iterations. And I think the other side which we're going to see grow is just more people using it. And I think that's, again, when we said, well, let's talk again in a year. I'm really curious what use cases we'll have discovered in a year.

54:28So already, when did we launch Nexus? That's like two months ago. We already have seen so many different interesting use cases, which actually also helped us shape some of the internal implementation. So I think that has been a super interesting learning experience for us. And I'm personally really curious how that's going to continue going forward. So right now, I can just say super exciting times. It really is. I mean, I think it's fascinating how we're all kind of trying to rediscover how do we package data for agents as a primary consumer. And we've talked about a bunch. I think, honestly, the coexistence with the metadata and having that be key both on the definition and retrieval side, or like query and retrieval and how all of that tracks, like that's a huge step forward.

55:14And we talked about semantic layers. I'm hearing all sorts of people were talking about, oh, we need much better semantics for agents because people just put that in their heads, but agents need it right there. Like, so yeah, discovering these patterns of what needs to be co-located now and what needs to be described. It's a fun time. Yeah.