Knowledge Graphs as Agentic Memory with Daniel Chalef

0:00Contextual memory in AI is a major challenge because current models struggle to retain and recall relevant information over time. While humans can build long-term semantic relationships, AI systems often rely on fixed context windows, leading to loss of important past interactions. ZEP is a startup that's developing a memory layer for AI agents using temporal knowledge graphs, enabling agents to retain long-term contextual information. It was founded in 2023 and was part of the Y Combinator Batch of Winter 2024. Daniel Shaleff is the founder of ZEP. He joins the show with Kevin Ball to talk about the challenge of contextual memory in AI, temporal knowledge graphs, ambient AI agents, and more.

0:44Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:20Daniel, welcome to the show. Thanks for having me, Kevin. Yeah, excited to get into this. So let's maybe start. Do you want to introduce yourself and Zep and what you all are all about? Yeah, so I'm a software engineer turned founder, and this is my second startup and my first AI startup. Like many, we're just under two years old. And our focus is on enabling the agentic future. And the way we're going to do that, or the only way we can do that is to ensure that agents have the right information available to them at the right time. And so ZEP is a memory layer for agentic applications. And we focus on midsize companies in the enterprise.

2:02And it's a very exciting space to be in. So let's maybe define some terms here, because while some of our audience is familiar with all these different things, maybe not everyone is. So when you say the agentic future, what do you mean by that? I feel like agent is a term that different folks mean different things by that. So how do you define agent? Yeah, it is so nebulous, right? There is a definition for agents. It's just become pretty loose. And the way I think about it is that AI agents today have an LLM as their brain, but they also have the ability to autonomously interpret instructions and make decisions, which is a very important aspect of agents, and then take actions to achieve goals that have been set for them.

2:51So those are the three high-level attributes that I look at when I think about what agents are. When you dig a little bit deeper into it, though, there are several high-level components or several needed components for an agent to be able to actually do what I just described. And they have tools, for example, for taking action. And tools might be something like querying the web, so searching the web. For answers, a tool might be actually taking action in a line of business application. So generating an invoice. And the other important aspect is for an agent to be able to reason and understand what to do next and make a decision.

3:38They need to have broad understanding of the environment that they exist in. So what is the user's world if there's a human in the loop? or what is the business world if they're purely reacting to changes in business state or our personal state, like our home. And they're going to have to have a very broad understanding of that. And so memory is important. So being able to recall what they did in the past and being able to then plot a way forward based on this new, maybe new stimulus that they've received. And memory really enables planning. And that's the other big, important aspect. And there's something called a perception action loop when we talk about agents.

4:27And so agents look at stimulus, so maybe a human's request or some event that is coming from their environment. They look at their memory, they perceive this, and then they decide to take action based on that using the tools that they have. So that's just a very high level view of the way I kind of look at what agents are. That's really helpful. So let's maybe dig a little bit deeper on what those pieces are. Tools, I think, is not going to be the big focus of what we're talking about here, but maybe first defining a little bit, that's essentially function calls, right? Making action using a computer somewhere.

5:04Exactly, yeah. Okay. And then reasoning, LLMs, like we're all kind of familiar with what these things enable. Memory, that's where you are focused. And I feel like it might be worth us kind of defining that a little bit more carefully. Is this just context that you're dumping into your LLM prompt in some way? Like, what do you mean by memory? Yeah. So I like to think of memory as quite expansive. and there are multiple types of memory that we look at when we think about agents. There's a short-term memory. A short-term memory is merely what is happening in the current, if you're talking about a human-in-the-loop agent, what is happening in the current conversation?

5:49What has the user just asked? What did they say previously? So what do I know? You can think of it as a human's short-term memory. But we also need long-term memory. And if you think about how the human brain works, long-term memories are often processed in some way, and they're put into a big data bank or memory bank for recall. And there are various different types of long-term memory as well. We could remember how to do things, so there's procedural memory, but then there's also semantic memory, where we build relationships between different events in our lives or things that we've perceived and connect them all up.

6:35And those go into the databank as well. And so those are various types of long-term memory. There are more, but those are kind of the big high-level aspects of long-term memory. And if we double click on semantic memory, this is where things get really challenging because humans have an uncanny ability to draw connections between various parts of our lives and our experiences. Sometimes we get it wrong. I forget people's names all the time. I forget what I ate for breakfast. But most of the time, we're able to file it away and recall it over very long periods, which is a marvel. But agents are going to have to have something that approximates that to be able to work effectively.

7:25Not only that, but the promise of AI is the ability to process vast amounts of data and make sense of it far more than humans are able to conceive or understand in a very short period of time. Okay. So that makes sense. And I think I understand why you double clicked on that particular area, because that ties into this concept of a knowledge graph, which is, as I understand it, what you all are building. So let's maybe talk about knowledge graphs. What are they? Is that this is the implementation of those semantic links? Yeah. So knowledge graphs are data structures that allow you to semantically model complex relationships.

8:09And they typically contain something called a triple, where you have two entities or what are called nodes in the graph. And those are things. So you could say a person is an entity or even a concept could be an entity. So you have two of those. And then between them, you have a relationship. And this is called an edge in the graph. And the relationship describes what an entity-to-entity relationship is about. So that's what the edge does. So that's a knowledge graph in a nutshell. And why they're very useful from a memory perspective is that we can build very dense and well-described data sets, semantic data sets, that are a very good fit for retrieval.

9:07So graphs allow all sorts of interesting approaches to retrieving data. There's actually a very mathematical approach that one can take to retrieving data from a graph and traversing the relationships that are available. And there are all sorts of wonderful algorithms that have been developed that allow you to interrogate the data and also do so in a very intuitive way. So I absolutely love knowledge graphs. I think they're an amazing data structure for condensing information. The challenge with building knowledge graphs has always been defining the ontology or the types of things that you care about and how those relationships between the things can take effect.

9:58So what sort of relationships are allowed in a knowledge graph? Defining ontologies has always been a very challenging endeavor. And then also building the knowledge graph has previously had to be done manually or in some sort of not very flexible way. But with LLMs, we have an amazing new opportunity to interpret data at scale and extract entities from unstructured text or even structured data and understand their relationships in ways we've not been able to do before. And again, at scale, because previously it was hand-building knowledge graphs. So let's maybe dive into that. What does it look like to, let's talk about ontology, right?

10:45Right. So defining your categories of things. When you say with an LLM, you can just sort of automate that away. Is the LLM defining your categories? How are you thinking about that piece of this? Yeah. So let me speak to Zep's open source graffiti library. What's unique about Zep's graffiti library is that it has a temporal dimension, but let's not dive into that yet. What I will speak to, however, is how graffiti is able to build an ontology on the fly for data, which is very, very useful when you're working in a domain that isn't necessarily as contained or bound as we've seen in the past. So if you are building agents that interact with humans, humans say the darndest things.

11:42Yes. So do LLMs for that matter. You get all sorts of bizarre things coming out of them. Well, yes. Yeah. And so if you are, for example, trying to understand human preferences as an agent that maybe controls a home, building a well-defined ontology can be extremely challenging. And that's one of the things that has encumbered developers in the past. It's very challenging to fit your data into an ontology. And so graffiti builds this on the fly by doing named entity recognition across the data that it is seeing. So it is determining what the things are and then intelligently understanding those relationships.

12:28Now, that could become very, very problematic at scale if you're not being clever about deduplicating the things. So these two things are alike. They might be differently named, but conceptually they're the same things. So we should use the same type. Secondly, these relationships are very similar. And so we should use the same labels for the relationships rather than generating an entirely new label. Because otherwise your graph becomes very difficult to query into if there are many different types of entities that are very much related. So let's maybe talk about type, because I think one of the things I saw on Zep's website is you talked about pulling out strongly typed data from raw chat information.

13:22how do you think about that type extraction? And are those types themselves then morphable? Because I might have a conversation where I name three things about this entity, and then sometime later, the agent is having another conversation and discovers, oh, there's a whole slew of additional data that's attached to this type of entity. Yeah. So I spoke a little bit about Graffiti's organic development of an ontology. Graffiti also allows developers to define types or custom types that are well described, i.e. developer can specify what the type is about. So this is a person and a person does X, Y, Z, or this is a customer and a customer is defined as follows.

14:12And a customer has a company name, a first name, a last name, and several other fields. And that takes things to the next level in terms of being able to have a well understood ontology. And you can do so very simply using Pydantic. Graffiti is written in Python, and you can do so very simply using Pydantic models, which is something that developers have become very accustomed to when working with LLM APIs, such as OpenAI's structured output. And what that allows developers to do is build an ontology that is better structured for their particular use case. But Graffiti will still extract entities that don't match existing types.

15:01And so that allows you to get the benefits of this organic development and well-maintained ontology, but also have structured types that make sense for your particular business. now if you have for example a user type that you've defined and graffiti is interacting with an end user of some sort and it discovers oh there's a whole bunch of other contextual information that keeps showing up around users that it thinks is probably useful does it create a new type overlaying users does it extend your defined user like how do you think about this sort of like the fluidity of what you're interacting with people when you're dealing with the LLM extracted things, like it can come up with all sorts of stuff.

15:50Yep. So it would likely create associated entities with the user entity that describe an additional relationship. So for example, in the company type that I described earlier, if we hadn't put first name and last name of a particular contact at a company, it might create a contact entity type and that contact entity type has a name that makes a ton of sense so you have a set of types that you've defined as a developer that you are pretty guaranteed these are going to have these fields in this way and then there's a set of types that graffiti is interpreting creating on the fly relating to other types if for example it's a graffiti owned type and it discovers here's some new fields will it modify the underlying type or it will do the same trick of like okay this is a contact and now we have a contact phone tree because we know there's a set of phones or I don't know what the additional entities might be.

16:49Yeah, it would probably, a thing would be a phone number. So a phone number would be a thing, would create an entity for the phone number and relate it to the user or the company. So it's pretty clever that way. You mentioned changing data though, which is where graffiti really shines. And I hinted at that a little bit earlier with graffiti is a temporal knowledge graph. And it's a little bit of a mouthful where graffiti is specifically designed to deal with temporality and dynamic data. And this is very unlike other RAG frameworks. And in fact, I like to think of a major shift coming in terms of how we view building LLM-based applications.

17:47Well, it's actually here already. And that is we've shifted from these Q &A chatbots, so question and answers over a document corpus, powered by semantic databases providing RAG and RAG frameworks. RAG frameworks work really well with static data, but the way I describe agents and the way I strongly believe agents exist in this new world is in a sea of dynamic data. And I think we're in a post-RAG era now. Look, RAG didn't last very long, three years, but we're in a post-RAG era now and graffiti is a post-RAG framework. It deals with dynamic data and it deals with or supports a constant stream of unstructured or structured text, structured objects like JSON.

18:39And it's able to integrate this data into the graph in a way where it is understanding whether the new data or knowledge that it has created from the data is in conflict with existing knowledge in the knowledge graph. So for example, just a very stylized example, I purchased a pair of Adidas shoes six months ago from an e-commerce agent and the shoes fell apart. Sorry, Adidas, I shouldn't have actually used the brand. And I send them back to the return system, using the return system. That's not part of this particular agent that I spoke to. It's a different agent that manages returns. and I sent a nasty gram back.

19:30You know, I'm very upset that the shoes fell apart. And so my previous brand preference, because I'd had a conversation with the e-commerce agent and said, hey, I love Adidas shoes. So my previous brand preference was noted as being Adidas. But now I've sent these shoes back with a nasty gram saying, I'll never buy your shoes again. I'll never buy Adidas again. My brand preference has changed. And so when we integrate that knowledge, stream of JSON from the returns agent into our memory store, we've got to update that brand preference. And that's what graffiti does. It understands that the brand preference has changed and that it needs to invalidate that previous relationship where Daniel loves Adidas shoes to Daniel loves Adidas shoes, fact is no longer valid.

20:26So this is interesting. And there's a bunch of different pieces we can explore on this. I guess, first off, just like at a very vanilla layer, under the covers, does this essentially look like timestamps on any piece of data of when it was invalidated or when it became known and when it became invalidated? Or like, how is this implemented? So graffiti, and I'm going to get a little bit technical here, has a bitemporal model. And it has both episodic memory and semantic memory. And we're actually, in Zep's implementation of graffiti, we've also implemented procedural memory. So episodic memory is an event or a chat message or similar, and you send it to graffiti, and it has a created date, the event that you created that episode.

21:23And you can think about it as, you know, the episode might be a conversation that we're having today. It might be a single utterance in the transcription of this particular conversation. And it has a timestamp, and that becomes the created timestamp of any sort of relationships that we add to the knowledge graph. But I also might have mentioned that I purchased a pair of Adidas shoes six months ago. And so there's two entities there, Daniel, Adidas shoes purchased, and that relationship has a created date from today, but it has a valid at date from six months ago. And then when we now send the shoes back and we say we're really angry, we create a new episode.

22:14That's a JSON data event coming from the returns agent. And we update the knowledge graph by integrating this new information. In graffiti parlance, there's another date that we can add to the edge or relationship. The fact that Daniel no longer loves Adidas Shue, so we can invalidate that, which is an invalidat date. And this is how we capture the time dimension of changing state. And it allows graffiti to reason with the new event data that it receives. So I love this distinction between semantic memory and episodic memory. When you put those timestamps in place, are those also a timestamp and a link to the episode that resulted in this change?

23:06Okay. Yeah. Okay. Interesting. So in the graph, you have an episodic node, then you have entity nodes that are related to that episodic node. And so what you end up with, it's really interesting. You have multiple episodes that link to the same entities. And you can see longitudinally over time, state changes. and very importantly graffiti allows your agent then to reason with state changes. Oh Daniel used to love Adidas shoes but he's back again and he wants to purchase some shoes. I can still see that he is a roadrunner, pronates and has wide feet but the fact that he loves Adidas shoes is now invalid so I won't recommend those to him and he just mentioned that he thinks he wants to try out the Puma shoes.

23:57So we can add another entity to the graph that there's potentially a preference for Puma shoes. Yeah, yeah, yeah. Really interesting. Okay. So next question related to this is how do you conceptualize like multiplayer or things like that? So like in this case, let's go back and just say you haven't sent the nasty gram yet. So we know So Daniel said, Adidas is great. I love Adidas. Maybe Kevin said Adidas is terrible. Is that tied to each individual? Like, how do you consolidate knowledge? And is there a way in which, like, if you have specific user graphs, can they be consolidated or shared in some way across a set of users or an organization?

24:38Like, how would you deal with those layers? Yeah. So in graffiti, you would, in the data that you provide, ensure that graffiti understood that there was a different speaker or the data was related to a different user. You could do that in the JSON if you're passing a JSON event in, or you can do it in unstructured text in a transcription format. Maybe this is a good segue to speak to what ZEP is versus graffiti and how things work in ZEP. So graffiti is a framework for implementing temporal knowledge graphs. It can be used as a memory layer, but it's a very generalized framework. ZEP has first-class support for projects, users, sessions, or chat history threads, role-based access control, data governance, and privacy functionality that's all layered on top of graffiti, plus SDKs for Python, TypeScript, Go.

25:45So it's an end-to-end memory service that is framework agnostic, and we have the ability to implement ZEP within LandGraph agents, within Autogen, or without any agent framework. Here in ZEP, you can have user-based knowledge graphs. And by default, if you have single user agent interactions, it would go into a user knowledge graph. And it's well contained and ensures that user data is managed appropriately from a privacy and security perspective. But ZEP also has the concept of group graphs that can be used for multiplayer scenarios. So for example, you can stream Slack messages into a group graph and query the group graph independent of users, specific users.

26:43So get results back from multiple users. And so there's a lot of flexibility there. So my answer is yes. Let's maybe talk a little bit about then what this looks like from a developer standpoint, because I think one of the things that was interesting to me was this idea about like, yeah, automatically inferring, changing things like, Like, what am I sending to Graffiti or to Zep? And what do I get back? How do I query? Like, what does that actually look like for a software developer? Yeah. So in Graffiti, and I'll speak to Graffiti because I love the fact that it's open source. It's easy for developers to implement as well.

27:21So let's talk about Graffiti quickly. Any unstructured text, JSON, or even transcriptions you can send to Graffiti. It's a very simple API. and you can create graphs on the fly. You can have multiple graphs in graffiti. So you can approximate what we do with Zep in terms of putting a firewall between users, namespacing your user graphs. And what it looks like on the query end is, let me speak to a design decision that we took. Developers don't have to learn Cypher or some other graph query language to query graffiti. There are two reasons why we did this. One, we think that we can provide developers with a very powerful search framework without requiring immersion in a language like Cypher, which is a common language for knowledge graphs originally developed by Neo4j.

28:28And the other design decision that we made is we're not going to have an agent or LLM in the query path because that's super slow. So GraphRag can take tens of seconds to get a response back. And GraphRag is that static RAG framework that is built around a knowledge graph and was developed by Microsoft Research. And it is slow because there are all sorts of things that it's doing in the query path with LLMs. And that doesn't really scale in production. It means that you can't use something like Graphrag for voice agents. And those are becoming more and more common. We don't want to be sitting at a keyboard and typing.

29:13We want to talk to a device, for example. And so Graffiti offers a number of different ways to retrieve data. and we do so in a way that's super scalable. So the entry point into a graph is typically through an index, not through a graph search query where you have to then traverse the graph to find things that you're looking for. And these indexes are semantic and full text indexes. And so what we've done in graffiti is every time you add new data to the graph, I mentioned you have entities and you have edges between those entities. We place a fact onto the entity. Daniel loves Adidas shoes. And that gets indexed both semantically and from a full-text perspective with BM25 indexing.

30:10We also generate summaries for those graphs, which is for the entities in the graph. And those summaries basically are summaries of all the relationships that a particular entity has. So Daniel loves Adidas shoes. Daniel pronates. Daniel has wide feet. Daniel is a roadrunner. And we index the summary. And so that allows you then to search edges and nodes semantically and with full text or both. And your entry into the graph is in near constant time as a consequence. and so you're querying subgraphs so you're querying subgraphs rather than the entire graph got it okay so let me make sure i understand this so first data entry you throw over your stream of messages or documents whatever this doesn't have to be fast so this is when you use an llm to do extraction or do whatever other things that you're doing there it's done asynchronously yes done asynchronously you can do a lot of essentially pre-compute of all these different things and some of you pre-compute a set of summaries, you put things into an index, and the index also then points to the nodes.

31:21Then query time, you go straight to this fast index, it loads up for you a summary of a set of things that you can throw into your agent's context right away, and a sub graph if you want to do more searching that's there for you. Yep. Well, it's not just the summaries, it's also the fact on the edge, which is pretty well defined. And then you can also do graph search based on that. So things like retrieve the nearest nodes to the node that was a hit. So using something called breadth-first search, which allows you then to get a more comprehensive view of the relationship that you have retrieved from the graph.

32:04So you can see adjacent entities and their relationships. And that's done explicitly? Like I will get back and I can then do this? Or is that done as a part of that internal first query? So graffiti has a number of different recipes that have been developed that are kind of like best practice pipelines. But you can also create your own pipelines. And the types of things that you can do are pull together BM25 and VectorSearch and use something like Reciprocal Rank Fusion to join the two search results and then use a LLM re-ranker to re-rank the results. Or you can use, we have graph-based re-rankers as well.

32:50So you can, for example, re-rank results from distance from a centroid node. So if you want to get Daniel's specific facts back, you can say, re-rank all the facts by semantic. Well, looking at the semantic search results, re-rank by distance from the Daniel node. And so these are all baked in as simple recipes in graffiti. And they're super powerful. In Zep's implementation of graffiti, where we run our own embedding and re-rank her services, our own GPUs. You can search a graph in under 200 milliseconds. And the most expensive pipelines have a P95 of 300 milliseconds. This is super cool. I'm doing some pieces of this type of work right now.