GraphRAG: Knowledge Graphs for AI Applications with Kirk Marple - #681

0:05All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Kirk Marple. Kirk is CEO and founder of Graphlet. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Kirk, welcome to the podcast. Yeah, thanks so much. I've been a longtime listener and glad to finally be part of this. I'm excited to have you on the show, and I'm really looking forward to our topic. we're going to be digging into what you're doing at Graphlet, but in particular, the broad space of GraphRAG.

0:42Tell us a little bit about Graphlet and kind of how you're approaching RAG as a space. Yeah, for sure. I mean, we've been around for about three years and started really building an unstructured data platform for everything. I mean, multimodal data, documents, audio, video, and really started getting interested in the knowledge graph side of this, of pulling all that data into a knowledge graph to make it explorable and then sort of saw the integration of that with RAG kind of come out over the last year and how we can benefit from that. We've known each other for a bit now. You attended our first TwiMLCon conference in San Francisco.

1:22That was in 2019. You started out doing, really trying to go after applying ml and ai to media talk a little bit about how that led you to the way you think about the problem today yeah i mean a lot of the problem in rag is is the r i mean the retrieval side but you have to have you have to have the data to retrieve in the first place and so we started focusing on i mean there wasn't at the time and still kind of isn't a five trend for unstructured data. I mean, that's kind of what we were looking at of where is the data pipelines to make the data available to AI models, ML models. And so we started at the ingest side, pulling in really all sorts of data.

2:12And in my background, I had a company in the broadcast video space. So dealt a lot with file-based workflows for that. And there's a lot of parallels of, I mean, pulling in data, running NLP on them, running computer vision. And so we kind of started with the content first of how do you get data into a system like this? And then search and retrieval, obviously, is a big step. And we start a lot with the metadata. I mean, how do you retrieve data? Well, how do you store the metadata first? And then how do you retrieve it? So like metadata filtering is common now in vector databases. So we were already doing a lot of that a couple of years ago.

2:49And as RAG kind of became the concept, we realized, I mean, we already have a great retrieval system to plug in to LLMs. And we already had solved the sort of media side of it that plugged into multimodal models very well. So that's when we kind of really focused on Graphlet as a platform that anybody could build an application on. And that's really what we've really been pushing for the last year. And so graph rag is a concept that grew out of a paper that Microsoft published some time ago. To what degree is your system like trying to implement the specific approaches from that paper as opposed to the general idea of applying – using graph – graphical relationships in a rag model?

3:47It was really interesting to see that paper. I mean, we'd been doing a lot of that already and just hadn't been talking about it as widely. But I mean, the first part of it is just how do you build the knowledge graph? And that's what we had focused on first of doing entity extraction, people, places, things and creating that from the content. And as we kind of built up our RAG system, pulling in that data from the knowledge graph. I mean, essentially, GraphRAG is really where we're at now. And that's what it was really. I mean, it was good to see it in a paper and kind of see other people kind of looking at that because it's something I've been thinking about for a couple of years.

4:24And really, honestly, a lot of this started with a podcast discovery platform that I was trying to build six or seven years ago. And it was a little too early. I mean, and now I know there's a there's a bunch of great project projects out there and transcription got cheaper. And it's just so much easier to build a platform like that. Let's talk a little bit about that entity extraction. Where did you start with that? How has it evolved? Is it a solve problem? If someone who's listening wanted to go about doing that, what are they going to find that's difficult? Let's dig deep into that part of the problem.

5:01What we found is there's a lot of overlap with the retrieval side of text extraction, text chunking, pulling those pieces together to do entity extraction. and really any NLP. So you kind of have to solve that part first and then use a model. I mean, we've used like Azure AI text analytics. I mean, we've actually used LLMs for this. And I mean, instructing the model to identify people, places and things. But what they're also really good at is identifying things like places, address extraction. We found LLMs are especially good at that, which was really difficult with NLP. And I mean, with with original algorithms.

5:43And so that I think that extraction side happens during the ingestion pipeline. So what we look at is. I've heard a little bit of the contrary, in particular, that large language models aren't great in a lot of cases for kind of traditional named entity extraction and that the traditional models are still better and maybe more controllable. Like, how do you think about where to apply what? We've actually seen that as well. I mean, we can use a mix. um we've seen the we've seen lms work great for specific cases like um for events like we were working with a community website where they were wanting to pull out like okay when when was the dance concert and when like where was it um we were able to guide the lm really well i mean to extract a an event um but i've seen it in other ways where kind of people and companies the classic sort of AI text analytics, I mean, like Azure, works way better.

6:49And so we can support both. So during the same ingested pipeline, we can actually run both models. And we can instruct it to say, okay, places use GPT-4, people and organizations use Azure, that kind of thing. You would think you can give an LLM a paragraph of text or a document and say, okay, you know, produce a JSON document that has all of the people identified and it just doesn't work as well as you might want it to work? There's cases. I mean, it definitely works in some situations, but what we found is, I mean, you get more noise in some cases. And so I think it's something you kind of want to try and look at.

7:29I mean, we've definitely seen a mix. I mean, there's some cases where, and I mean, honestly, the Azure type, I mean, or Amazon type models have similar noise problems where it'll identify a term as a company and it's not really a company. So there's a data quality issue that kind of feeds into both these sides. And I think that's where it does take a bit of testing and evaluation to get it working right. And have you been able to identify specific patterns that characterize the cases? One thing we've looked at, we haven't released yet, is sort of a chaining model using sort of an NLP style to identify the entities and then go in and refine that.

8:15We do support data enrichment today where when we identify an entity like a company, we can go call out to like Crunchbase or Wikipedia and enrich metadata for it. But one thing I really want to try is once you've identified the entity, go and re-identify and prompt the LLM to say, hey, I think this is Microsoft's in here. go, I don't know, grab everything you can find about Microsoft. That's an area that I want to explore definitely a bit more. So you've got these multiple methods to extract the entities, and you mentioned that that is part of the data ingestion. How is that used in the context of ingestion?

9:02So what we call our content workflow is kind of a multi-stage. So ingestion is really just like downloading it from a blob storage or a website. And then we have a stage which we call preparation. And that basically includes audio transcription or text extraction, PDF extraction. And what we get out of it is a canonical form of, I mean, essentially we store a JSON file of here's the transcript or here's the text with semantic chunking or page chunking. And then we have an extraction stage. And that's where this would happen. And so we kind of have this sort of state machine that the content goes through.

9:42And in our platform, you can configure each stage of the workflow. And yeah, at extraction, I mean, you can tell it which model or which API to use, what you want out of it. And then we basically take the results of that and feed that, connect everything up in the knowledge graph, basically. And then we have an enrichment phase after that, which is optional, where you can kind of just say, oh, I've identified, I mean, a company now go get its address, that kind of thing. And you recently joined our generative AI meetup. You've been involved in our community and kind of talked a little bit about this and did a demo.

10:22And one of the contexts or one of the questions that came up in that conversation was like RDF and these entity relationships, you know, from the traditional NLP world. Do you use those kinds of relationships in the graph that you construct or is it more ad hoc? That's a great question. I mean, the way we currently do it is we're basing all of our entities on schema.org. So JSON-LD kind of style. So we're not inventing our own data model for that. And I think that's important to have canonical kind of reusable data. And the LLMs actually know JSON-LD really well. So we found. And so that's kind of the first part.

11:09I mean, today we're not doing, we're leaning more on entity to content relationships rather than entity to entity relationships. That's most of what, I mean, we could do both, but I think that's mostly what we're trying to see. Because with the entity to content relationships with GraphRag, you can then say, oh, for this piece of content, what are all the entities related to it? And what are those entities related to in other pieces of content? um and so that our our graph leans leans a bit more on that on the entity to content relationships okay okay and so there's less of a need or desire at this point to kind of trend traverse entity to entity relationships to pull in this broader graph of content it's more kind of two-step as opposed to a fan out and that's what i think um like yohei has been working on some graph things you might seen on Twitter and there's been different kind of open source projects that are a bit more like, I mean, I work at X, that kind of relationship.

12:09And I mean, we could definitely do that. I mean, but our graph just because of what we're using it for is not leading that direction today. So you were talking through your workflow and you kind of got through the ingestion part of the pipeline. What's next? The preparation stage and the enrichment stage and all those, essentially end up with data in a vector database and in a graph and a document store and so what we do is have a sort of a hybrid data storage model where um and object storage we can leverage as well and so i mean if we're we're caching things essentially there and so we kind of use that all together where the graph is kind of replacing the relational index and that's the relationship between all the content um i mean we have collection kind of patterns we have the entity to content patterns.

13:02And so this is something I kind of came up with over the last several years and just kind of refined. And that's worked really well for us where we can then walk from one piece of content, either via similarity in a vector, sort of that vector angle or through the metadata and the graph. And so we can kind of have that hybrid approach and essentially a searchable knowledge graph. What's been your experience working with the vector databases Do you kind of support all of them or do you have preferences? Are there like key features that you're using in your system that are, you know, not universally supported or are you just using the basic capability of the vector databases?

13:48Yeah, that's a good point is, I mean, we had looked at, I mean, all the kind of major players and talked to a lot of folks. There's a lot of great ones out there. we were Azure native today. So we're leveraging a lot of kind of Azure data services like for our graph database. And we were actually already using Azure AI search. I mean, then cognitive search for our keyword search. And right at the time when I was evaluating vector databases, they came out with their vector index support. And I tried it and it worked. And it's a nice because it supports metadata filtering and keyword, vector and hybrid all in the same box.

14:30I mean, we're not plug and play with vector databases like some other things. We're kind of more, we're going to pick different things connected together and have a managed platform on top of it. And it works great. I mean, and they even just increased their performance and updated their storage and things like that. And we're actually going to be at Microsoft Build next month kind of showing this off. And I mean, in that Microsoft domain, but it has worked great for us. And we could, I mean, theoretically swap it out if we wanted to. But we've actually had some really good experiences so far with this one.

15:09And so we're kind of broadly seeing, you know, as RAG and vector databases become more popular, vector becoming kind of a layer on top of, you know, a lot of different data stores. So I just came back from Google Cloud Next and they did a similar thing where, you know, BigQuery now has a vector layer. AlloyDB, which is kind of their Postgres, now has a vector embeddings layer. My question is, do you see like the same thing, you know, happening with graph as well? Like, is it going to end up, do you see us moving towards a converged world where, you know, you put your data in and you have all these different kind of semantic abstractions on top of it, you know, where you're not replicating the data?

15:58Or do you think that, you know, graph is, you know, so distinct that they, you know, remain disparate systems? it's a really interesting point i mean it's something i've thought a bit about because when i was first looking for kind of the magical database that i really wanted like it didn't exist and i had to kind of create this this franken database from from a couple different things um the searchable side like keyword search in a graph i know that a couple vendors have have solved and in that world i would envision the vector comes next um i think there's it could I mean, you could integrate something like that with the graph.

16:40I mean, the big problem we had, honestly, with graph databases is the payload. You can't store a ton of data in a node. And that was a big reason that we kind of came up with this way that it's sort of a three-tier storage model where the graph is the index, the heavy metadata is in a JSON store, and then the really heavy content is in the file system, in object storage. And so we kind of used that sort of HSM model of, like, layered storage. And that's worked great. I mean, we only kind of pull it off desk if we really need to. And but to your point, I think I don't think it's a done deal that it'll go there.

17:17I mean, I think we'll have to see. But I wouldn't be surprised, really, if they start. I know they've started adding the keyword search, so I wouldn't be surprised on Vector. And along the same lines, like I think, you know, Vector databases have been around for, you know, quite a long time. And we were making progress in kind of shifting from keyword-based search to more semantic search and embedding-based search. And most of the Vector databases, vendors that we see now kind of started as these tools to support the search use case. You know, RAG came and kind of popularized that whole space and, you know, Vector then started become getting pulled into, you know, every data store.

18:06Do you think that GraphRag will have the same impact on graph? Graph databases have been around for a very long time, like longer than vector databases, I think, right? Neo4j, I've known those folks for quite a long time. um you know they've always had their place but they've never like i think had the the vector moment like vector databases recently have do you think that that you know based on what you've seen with graph rag like do you think graph uh graph rag is that killer app for graph databases i think it's definitely possible i mean graph databases i've i mean it has been one of those sort of background things where it's really useful for some specific use cases a lot of times the algorithmic side of like i mean running i mean algorithms heavy algorithms on a very large graph um i think the idea of more of a property graph and this sort of entity graph and graph rag i think could be a way that it's like what happened with vector and rag i mean there are several vendors that like made themselves like that that created huge companies out of just that that plug and play so i think we'll have to see i mean the the sort of value of graph rag is still a little TBD.

19:27I mean, we're still exploring it, but I think that's what makes it interesting. I think there's a lot of interesting paths you can take to see how to get the sort of squeeze of value. And I mean, I'm optimistic. I really think that, I mean, more people putting these relationships in the graph, just rather than just a typical relational database, gives them more opportunity to find the sort of explore their data in new ways. You go through this ingestion process that populates your kind of four data stores um you know interrelated data stores and is that the end of the the pre-processing uh step and then the um you know then we're to the next half which is what happens at the time of a query pretty much yeah i mean other than enrichment i mean which is which is optional i mean once the data is in the data stores in the vector index i mean that's like i mean if you call our api and say hey ingest you are like a url then the footprint becomes okay the extracted text and the and the data in the graph then it's all about retrieval and then it's you can make a query and that query um basically can be a mix it could be keyword only it could be vector hybrid um the metadata we'd already been doing a lot with metadata filtering because one of our key sort of thesis points was index everything in time and space because we were actually doing working in geospatial as well so a lot of our metadata is not just title author keyword it's i mean what lens on a camera are you using what's the gps location of the the the image or the video so like we have we have a really thorough set of metadata that we can extract from any media and that's when it becomes really interesting i think for i mean metadata a driven reg.

21:20I mean, asking questions about locations, asking questions about time ranges. And then the graph reg is really pulling out those entities and asking questions about, okay, well, where are these entities mentioned? And so that everything kind of drives towards that retrieval model. It strikes me that there is a lot of potential complexity in like the fusion of these different, you know, retrievals. You know, with, yeah, I think folks who listen to the podcast have heard me talk, you know, previously about like, it's easy to get a rag demo up and running, but to get something that's really, really good into production is difficult in part because there's a lot of tuning that goes into the retrieval.

22:07and that's just with kind of one you know data store with just your embeddings and and you know chunks and all that kind of tuning all that can be difficult now you're talking about layering in you know metadata and graph and you know concepts like re-ranking like what does re-ranking mean now across three different you know sources of of information? Like, how do you approach all that? I mean, we sort of take a layered approach. I mean, retrieval, the first is this kind of search that is, it definitely does touch. I mean, it touches the graph, touches the search index, the vector. And so the retrieval kind of step happens first, but it applies to kind of that triad of data stores.

22:52And then we actually just support added support for the new Cohere re-ranking model recently. And so what we do is the output of retrieval gives essentially a ranked list of sources. And but usually the sources has metadata that we pulled from the data stores. And then we provide that to the re-ranking model or do it ourselves. When you say a ranked list of sources, meaning source documents or sources like your three different metadata vector graph? content sources like document chunks or document sections, things like that, or chunks of an audio transcript and things like that. So a source is kind of an abstraction for like, hey, here's a piece of text that we found probably in some content.

23:37Got it. So you somehow execute, you know, a query across these multiple systems, you get back a bunch of chunks, and you treat them, You know, from a re-ranking perspective, you treat them equally like you're just doing the best you can to rank them based on the content of the chunks as opposed to, you know, the context in a graph or the, you know, the neighborhood in a embedding space or something like that. At least today we are. I mean, I think that's something we're exploring. I mean, one is time, sort of time relevance of sort of doing time clustering. I mean, we're looking at geo clustering.

24:21We're looking at some things like that. The graph can help with that as well. I mean, today, I mean, what's in production today is essentially just the content sources getting re-ranked and those content sources can come from retrieval. um i mean the the big thing we're we're doing today with um we we do support retrieval strategies for like expanding um the the chunk into a semantic chunk in which is like a section of a document or i mean chunks of a of a transcript and things like that so we we call those strategies that then that's where it's the knobs people could turn during during the retrieval step and then that's where we just added added the re-ranking strategy which initially supports cohere um and that's really helped i mean we can definitely see is the relevance you get out of your vector database or even the output of the search it's not i mean that's why these models exist i mean it's not exactly in the order you're expecting and lms can adapt to misordered data but it's the filtering that i've found is throwing away the irrelevant data um lets us kind of have a filter, kind of a low-pass filter on, I mean, okay, let's just get rid of stuff that doesn't make sense and that could confuse the LLM.

25:35Maybe this is a good time to introduce the topic of evaluation and how you think about relevance and quality of a retrieval. Yeah, I mean, a lot of it, I mean, everybody kind of goes through that ad hoc phase of, okay, it looks good. And originally, yeah, vibe check, exactly. I mean, and we started there. And we've actually just started working with a vendor in this space that has kind of more automated evals. And hopefully, I mean, we're looking maybe by next week, we'll have some details on that. But yeah, we're starting to kind of lean more into that. Because, I mean, for me, I'm not a data scientist.

26:15I came from a more traditional software background. A lot of this is like unit testing and integration testing. I mean, and you have to have suites of these tests to run when a new model comes out or, and I kind of look at it in that way. And so we are trying to automate that more, more than just VibeCheck. And I think there's some good vendors out there and good projects that really help with that. Are you thinking about dynamic optimization, DSPY and those kinds of things? You know, dig into that whole prompting space. Yeah, I mean, I would call it, I mean, what we do is more of a prompt compiler.

26:49and so it is dynamic prompting we're not just using i mean there are there are static phrases that we use or like i mean paragraphs that we we use for like some of the instructions and guidance um one of the things i came across and this is actually when i started implementing cohere months ago though how they like xml i mean the the cohere prompt really like xml and i was actually able to realize that um opening eye likes it as well and pretty much on most of the other models So we have sort of an XML template that we sort of compile to that has different sections. And it has a context section at the top and instructions and guidance.

27:28And so we've kind of broken it out into, okay, here's how I'm going to guide. Sort of guidance is like what not to do. And instructions is kind of what I want you to do. And then we compile the sources and then the user prompt piece at the bottom. um we've had to do some things like dynamic based on model where like i don't know like haiku um forgot the schema the json schema like it you had to remind it of it at the top and the bottom i think it was and i've seen that with a couple different models and um and those are really some things that it that it lets us be dynamic on the fly basic so there's a there's a structure but it is kind of a compiler that we dynamically output based on the query, the incoming query and all the retrieval strategies and all that kind of thing.

28:19Is it an LLM that's compiling this into a prompt? Is it, you know, a set of rules or heuristics or a combination like? Yeah, the compiler is just code. I mean, honestly, it's just, it's straightforward. We are using LLMs for things like, what is it, a prompt rewriting. So we do have a way that we have a couple strategies for optimizing for semantic search versus kind of rewriting the prompt. We actually just have an experimental one for multi-query now that'll kind of break it into multi-queries, which actually works really nicely. And I know that, I mean, Lomindex and Limechain, they've seen those kind of results as well.

28:59Those work really nicely. We use LLMs for like summarization of the conversation history. so we support like a windowed conversation as well as a summarized history so those kind of things but the the compiler itself is kind of it's just it's a code i mean it's just kind of looking at all the context that it has and then generating that essentially a strength it just gets gets put into the llm and when you you said that the llm's like xml

29:31uh you know how different how much stronger is that statement than they like structure like they like that particular kind of structure apparently like as opposed to you know uh headings and paragraphs yaml json that for whatever reason xml is just like some magic pixie dust that makes them work better i mean i think it's it's a bit of both but i think i learned it from i mean it's It's just how Coher documents, they kind of say, hey, here's our preferred prompting format with and using XML to kind of create sections within that. And I found that when I backported that, because I tried YAML, I tried JSON, I tried a couple other things.

30:14And I mean, I don't know if I was a little surprised, but I mean, it's probably trained on a lot of data like that. But I was I saw how it we didn't have to change our method depending on the model as much. It was a common thread that I found that pretty much every model I've tried listens to the prompt very well when it's structured like that. Now, could I do that as YAML instead of XML? I think OpenAI is definitely more resilient, and you can give it more options for the structure I found. But Cohere would then have a downside, at least with their original models, where it wouldn't listen to the structure.

30:53So what we were looking for was something that kind of worked across everything and could just be kind of a standard approach. For a given use case, are you using multiple models and kind of orchestrating them and trying to identify what's best slash cheapest for a given step? Are you defaulting to a single model that you like the best? How do you think about the model space? Yeah, I mean, we let developers pick. We have what's called a specification. It's kind of a preset that you can pick your model, pick your tokens, like token limit, system prompt, all that kind of stuff. And then depending on what conversation they're having or what, like if they can use a different specification for JSON extraction per se than for having a conversation.