Agentic AI at Glean with Eddie Zhou

0:00Glean is a workplace search and knowledge discovery company that helps organizations find and access information across various internal tools and data sources. Their platform uses AI to provide personalized search results to assist members of an organization in retrieving relevant documents, emails, and conversations. The rise of LLM-based agentic reasoning systems now presents new opportunities to build advanced functionality using an organization's internal data. Eddie Zhao is a founding engineer at Glean and previously worked at Google. He joined Sean Falconer to discuss the engineering and design considerations around building agentic tooling to enhance productivity and decision making.

0:42This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:00Eddie, welcome to the show. Thanks, Sean. Great to be here. Yeah, absolutely. I'm really looking forward to this. So I'm fairly familiar with Glean. A bunch of my former colleagues at Google have gone and moved on to working at Glean. And I'm curious about how is the original vision of Glean evolved from this enterprise search company to now working and having a still a reasoning agent as part of their offering? Yeah, yeah, definitely. I'd like to think that our vision actually hasn't changed that much. And Enterprise Search was really just the first way that we could deliver part of that vision.

1:37And so the way I like to think about it is with Enterprise Search, we were meeting knowledge workers in a little slice of their sort of job to be done, their user journey, right? You know, they open their computer, they need to do something, whether they're an engineer writing code, a PM researching other products, a salesperson prepping for a call. They all kind of have some journey they need to do. and we were looking basically to make that journey more meaningful easier and sort of free them up to do more things right and so with search it was really about okay if this person can do the work of thinking what is my high level goal what do i need to do identifying that i need to find some information and then going and interfacing with a search product to get that information right so you've really sort of helped them in that little segment right and all we've done as we've evolved from clean search to assistant and now to this agent platform is sort of broaden that segment both to the left and to the right, sort of helping them need to do a little bit less work to understand what do they need to know to do this task, meeting them earlier on in that journey, as well as to the right, sort of helping them get more done after they found the information, right?

2:46Maybe it's just synthesizing the information on the result page. Maybe it's actually starting to help them do whatever they're going to actually do with that information, So I really see that sort of evolution from enterprise search into where we are today as an extension of the same vision we had to sort of help knowledge workers everywhere. Yeah, I think one of the unique aspects of Glean and how you've managed to position yourself in the market is with the models themselves, they're very, very smart, but they're really, really dumb about your data, essentially. They know all of this stuff, but they don't really know anything about you and your company.

3:22And that's where the challenge is to build actual meaningful applications. And I also think that's one of the challenges with some of the other agent offerings that are available in the market. It's like, it's great. This tool can be amazing, do all this reasoning capability, but we're essentially getting that data. And since Glean is already plugged into that from sort of the very beginning, has access to all the sort of really rich information that you essentially want these agents to have access to. Right, right. And I think that that's an important sort of nuance for people to understand as the underlying LLMs that power these agentic systems get better.

3:57Because them getting better is useful for everyone, but it's important to define what they get better at, right? They're not getting better at knowing your company's knowledge. That's a binary thing. Have they seen it or have they not, right? And so you're right that figuring out how to inject, we call it context injection, putting enterprise context at the right places throughout an agentic systems execution is really important to actually getting them to work. Yeah, because I've seen you have a really powerful model, but if you don't have good data, it doesn't really help. You can have essentially a lower power model that has access to the right data at the right time, but perform the best model in the world.

4:40So it really comes down to, I think, the central challenge for most companies building any kind of AI experience today is how do I kind of liberate the data and find that subset of data to provide stuff into that context window so I can actually steer the model in the correct direction to generate a meaningful response. That's right. This sort of notion of world knowledge and then company knowledge as a layer on top is something sort of tying back to your first question that we've been thinking about in the search world for a while. When someone comes to a product in their work life, a search bar, a chat interface, they're not sort of only in the company context.

5:17They're an employee of this company, but they've built on top of a foundation of world knowledge. And so we have to build our systems in the same way. So even in the search setting, if you think about embedding models used for retrieval, you still run into the same dynamic of, hey, have these models seen this new data or this data that's specific to the company? If they haven't, they're not going to perform well. So how do you sort of layer on that new understanding? How do you adjust these models to account for that? The same thing goes for all the generative products and the generative use cases in the agent world.

5:46How do you effectively acknowledge that there is a prior, if you will, in that world knowledge, but you need to augment it with what's there? And it's a really hard problem. I'm not going to claim we've fully solved it by any means, but I think we're really well positioned to keep making progress and building things that give value to people. So I think it's worth stopping down for a second and actually explaining from your perspective, what exactly is a reasoning agent? I think there's a lot of variation in terms of what people think about an agent. And I know Hugging Face recently came out with this nice framework of kind of like the levels of agentic AI.

6:21So how do you essentially define a reasoning agent? Yeah. And I want to preface this by saying, by no means do we think, you know, we're not incredibly opinionated here. I think there's a lot of framings and frameworks that are being developed and they all have their merits. I think we've sort of come to our own internal viewpoint on this. And even that continues to evolve because it does matter when you talk about, okay, if we have a reasoning agent and it has access to tools, right? What are those tools themselves capable of? And so one framing is, okay, a reasoning agent is something that can, given a set of tools, formulate a plan to satisfy an input, and then go and execute those tools, right?

7:06But then there starts to become many different questions, extensions, right? Can it use these tools in an iterative fashion? What is the granularity of these tools? Are the tools themselves other agents in the sense that they also have the ability to call out further into other systems? Or are they sort of more static? And so I know this is kind of a cop-out answer, but we're trying to not draw a hard line around this while still figuring out how can we keep our framework flexible enough to adapt to how the industry is thinking about agents and thinking about agents they might build in the open source world, they might build in other parts, and how can we make sure they can at least integrate with Glean as the dust sort of continues to settle on this.

7:49So when customers come to you and they're asking for an explanation between like, how are agents different than RAG, how are agents different than some sort of like AI application to follow the workflow? Like, what is sort of the explanation there? Yeah, I think for the former, in terms of how it's different from RAG, I think the main component here is that we would see agents as an extension of RAG in the sense that the only tool that a RAG agent, quote unquote, has access to, or the only flavor of tool is some sort of read or retrieve type tool, right? It can issue a search, whether it's a search engine like Lean or a federated search engine.

8:27And then it sort of generates, right? Retrieve augmented generation and it generates based off of that, right? And so agents are simply an extension of that where the content that's being generated may not be the response to the user. It might be the next step in a plan, right? And being able to sort of sequence this out so that it's not just a retrieve and then a generate step, but rather a more extensible. Perhaps it's multiple retrieve steps. Perhaps it's a generation that goes into the next step that is then another retrieval. It's providing access to more actions that are not just retrieve and read actions, but actually executing.

9:02It could be executing code. It could be interfacing. We call them write tools, but sort of writing out into the world. So those are some ways that we sort of distinguish agents from core RAG. And the second one you mentioned, what was the second distinction you were looking for? Essentially like an AI application that's following some sort of workflow. Yeah. Yeah. So I think in terms of the word workflow, I think the distinction between what people have been calling agents, and this is where terminology does get a little bit blurry, but we do like to think about things in terms of a sort of static and dynamic, right?

9:35So a workflow might represent something that's more of a fixed execution. It might be multiple steps. There might still be LLM calls that are allowing for some variability, but the actual execution flow of what steps will be executed is fixed, right? And the system itself, quote unquote, can't modify that graph, right? But once you introduce some level of dynamic where you say, okay, for a given query, we're actually constructing a graph or perhaps iteratively constructing it, that becomes the distinction between a workflow or what people are calling a workflow and something. And of course, you can take, you know, graphs that are constructed for a given query and freeze them into workflows that are repeatable, but that's another distinction between these sort of workflows and more dynamic agents.

10:21Right. So it really boils down to like, what is the control logic? Is it some sort of predetermined, pre-programmed set of steps, or are you essentially allowing sort of the brain of the agent to determine what that sequence of steps is? That's right. And they definitely both have their place. It's sort of a high risk, high reward, right? The more levers you give the system to create something flexible, the more powerful it can be. you could hope that it generalizes to new use cases, new queries, but at the same time, it can be more unpredictable, right? And if you want something more predictable, you want something frozen, you certainly can buy sort of the size of freezing that control logic, as you said.

11:00And of course, you can use a dynamic system to help you build the initial sort of graph that you want to freeze, iterate on it, and then freeze it and use it indefinitely, right? That's obviously another path you could take. Do you see most people using some sort of hybrid approach where some part of the sort of workflow might be agentic where it's a more dynamic set of reflection steps or something like that, whereas other parts of it is a more predetermined set of orchestration steps? Yeah, I think we do see both. And I think everyone wants a fully looping dynamic system, but they often find once they deploy it, it's a lot harder to sort of get a handle on.

11:41And so people end up building more constraints back around their system and sort of freezing different parts of it. But it really depends on the use case and the stakes. If you have something that are high stakes, you can't afford that execution graph to change. And for us, I think what's interesting, sort of tying this back to the enterprise case is when we talk about something like reflection, right? A concept like reflection, which is, hey, given what I've executed so far, you ask this central brain, this LLM decision-making logic, what should I do next, right? And the really tricky thing about many of these enterprise, not all, but many of these enterprise use cases is the same dynamic between world and sort of company knowledge, right?

12:23It doesn't know what it doesn't know, right? Even in a simple, I mean, this problem is present even in RAG, right? In a simple retrieve and then reflect before generate step. You know, if I'm asking some question, it could be as simple as about like my holiday calendar that my company has for this year. Depending on it, what the retrieval engine does, if it retrieves the right things or if it doesn't, how can you ask that reflection component to do the right thing afterwards, right? If it's sort of beholden to the performance of that previous upstream system, if you present it, here's the query that was planned, here's some results, you know, what should you do next, right?

13:00And you could imagine a case where the search engine didn't return the right results and it decides to respond. So how can you really, reflection, I think, is a very hard task to do in the enterprise context because these agents generally don't know what they don't know, right? And so that's a very careful context injection sort of problem to think about and work on. How do you deal with sort of the like unbounded execution, even outside of reflection, like, unless you're putting like a hard limit on how many times we can kind of loop over this, like, how do you essentially manage the fact that the execution cycle might be unbounded?

13:37Yeah, I think as simple as it sounds, the first thing you mentioned is kind of like the easiest way to do it is you might allow for a fixed number of executions at various stages, right? It could be overall the number of steps, you know, you can only create an execution graph with this number of steps, or within a given a number of steps, you can only have it iterate this many times. Those are some good guardrails to put in. I think it also sort of depends on the complexity here. There's probably some neat analog to, there was a paper last week or two weeks ago that people were benchmarking a lot of these thinking models like the LMs themselves and sort of measuring their performance as the number of thinking tokens increased.

14:15And they found that, you know, there's a sharp decrease in performance once the thinking tokens became too long. And the sort of, I don't know, intuitive analog is that, well, it's rabbit holing, it's spinning, right? And you could probably extend that same thing to agentic systems where, you know, if your number of executions, if you're looping too much, it's probably unlikely you're going to reach a good outcome. And I think there's a careful balance there, right? I think there's a lot of value that people can have with simpler agents that do execute a much smaller, you know, we're talking order of single digit number of executions rather than dozens or hundreds, right?

14:47And so I think playing in that smaller space still provides a lot of value and you can add some simple upper bounds depending on the use case. Does it help to think about essentially breaking up these problems where you want an agent to operate and solve some sort of tasks instead of using sort of one monolithic agent to perform that to split it up essentially into a multi-agent system? Totally, totally. And this goes back to I'm trying not to be too opinionated here, but personally, I do have some opinion in terms of, for us, we're also thinking about, okay, how do you scale, right? How do you both build agents internally and help others build agents in a way that isn't bottlenecked on a monolithic system?

15:27Again, drawing from ML systems and sort of first principles here, a lot of times, ML systems in large products become too monolithic and there's a downside to them because then, okay, you have your team of 15 people. Now they can only all work on this one model. And everyone's just trying to work on this one model. So your actual rate of improvement is lower than if you had left that factored into multiple systems, you could have multiple people working in parallel and sort of you're giving up that short term quote unquote gain for a more medium and long term gain, but you'll reach a better spot.

16:02And I think the same thing applies here where our internal approach right now is a little bit more, okay, yes, you do have a central agent, but the sort of tools, and I use that term again, a little bit loosely that it's given access to maybe other agents. And so if you can delegate more to those other agents, and it's not just about in the short term, what you can do, but someone full time focused on making that agent successful is always going to be more, you know, going to do have a better time doing that than you trying to solve all these things at the central level, right? So I do think there's a huge role for delegation here.

16:34And it's about figuring out that right interface between that central agent and these delegated agents. Yeah. I mean, I think one of the things I see people sort of missing as they sort of dive into the space and are excited about it is that as exciting as this stuff is, it really carries all the same challenges that you have with running any like large distributed system. And the scale problems aren't simply just about infrastructure scale. They're also about like, how do you scale the teams? And essentially, how do I design this in such a way that I can loosely couple these things, treat them essentially like a microservice that can kind of operate independently, even leverage different models and stuff like that, independent of some of these other systems.

17:12And the teams don't have to necessarily know exactly what's happening within each particular team and have these hard, fast dependencies between them. Totally. I mean, I know this podcast is software engineering daily. And so, you know, experienced software engineers everywhere, I'm sure have been punching the air, seeing, you know, LLM agent systems being designed without thinking about core engineering principles that just like all the things you said that are still incredibly relevant. And I think, you know, it's the ability for these systems to show you something really awesome in the short term and give a good proof of concept has made it easier to sort of forget good engineering principles when designing them for the medium and long term.

17:52And obviously the rate of change and the pace has made that hard too. Yeah, absolutely. And then in terms of like an agent, going back to like a singular agent, can you break down what the components or the anatomy of an agent is and how each of those are used in order to essentially come up with a plan, perform these reasoning steps, potentially execute tools and sort of to solve a specific problem? Yeah, sure. The way we're thinking about it is, you know, we have a central system that has access to a set of tools and, And, you know, its first step is to develop that first pass at a strategy or a plan using those tools, right?

18:29And how it does so is important. You can obviously give descriptions of the tools or if you're using function calling, whatever that interface may be. But as with all LLMs, you know, having good in-context examples is really important, right? And so this kind of goes back to the team scaling component of like, how do you influence this central system? Well, you can have teams that are building these sort of golden in-context examples that say, hey, I want to make sure that when this central agent sees a query like the one I care about or sees an input, that they can effectively sort of build the graph that represents what I want it to look like, right?

19:07And so I think that the first part is assembling this input to this main, what we call strategize call, that is composed of available tools, but also curated important examples that demonstrate how to use those tools, how to synthesize them in this multi-step way. And then it becomes the output of this system is a sort of graph that can be partially executed or fully executed. But again, that graph, it has some level of delegation. Each of these tools, the way we're thinking about it, you can defer to them more and more. I can say, hey, for this tool, I have an objective. I'm not telling you exactly how to accomplish the objective because that is the tool or the sub-agent's goal or not a goal, but it's their sort of domain is to figure out how to do it.

19:50And so you can sort of distribute the cognitive load, if you will, over the sub-agent as well. And then the sub, you know how you have different teams working on sub-agents, different folks can work on these sub-agents and make them work well. And then, yeah, so then you can basically execute your graph and sort of come back to the central system as needed. But that's kind of the rough flow that we're thinking about right now. Is there a limit to the number of tools that any one agent can handle? Definitely. I think there fundamentally is. And even if we're not even talking about hundreds of tools, even if we're only talking about a dozen tools, the permutations of chaining them together to accomplish some set of things is also obviously exponential, right?

20:31And so you do need, this is our kind of drum that we beat, a lot of it can come back to search. And for us, it's thinking about, okay, how can we make sense of all these permutations, especially the ones that are these sort of in-context examples that are examples of how to use these tools in sequence or in parallel. and take that eventual set of tens of thousands, hundreds of thousands, millions and search over them down to something that's much smaller and guide the LM to that, right? So you almost have like a graph search problem or just some sort of item search problem that you can then say, okay, now it's a tractable set of things that we want.

21:08We've given an assist to the central agent. It doesn't have to fully reason over all the possibilities. You can, again, factor part of that system out itself to a search problem, do that search separately, and then say, hey, here's five, 10, 20 different tools and or combinations of tools that are probably relevant for this, rather than, you know, the entirety of all the combinations. So the idea there is that, you know, I have some sort of context about what it is I need to, you know, maybe data I need to gather, then I can potentially, you know, perform the search. And then essentially, that's sort of my first line of defense as a tool at sort of the main agent that's operating here.

21:44And then from there, I know, okay, well, you know, Slack has some of this information, you know, Google Docs has some of this information, you know, Confluence page has some of this information, and then I can essentially limit the set of tools to those three tools to then go and execute against. Yeah, that's a good way to think about it. And another thing, you know, from our side, the way we think about it is the other dimension on this is the sort of company specific dimension, right? There are, you know, mostly companies kind of do work or store information in roughly the same way, but that's very quickly starts to break down companies develop their own ways of storing information.

22:17That's what we've been building with search, right? So even the sort of algorithms that you would use or the nuances of like, okay, what tools or workflows are relevant for company X for this query might be different for company Y, right? And so the way that you build that search algorithm can start to be, you know, it's not just about, you know, you need to do query understanding in the same way over, okay, when Confluent users are asking about this kind of query, what kind of workflows should be surfaced in this search setting versus if a Glean employee is asking that, it might be different, right?

22:48And that kind of goes back to, you know, a lot of what we've been doing in Glean around, not just around language adaptation, but also around these other signals that are important to understand, right? Like is, you know, Sean is working in these different places. So when he asks this query, again, that search algorithm should be actually personalized to him and it may return a different set of things, a different sort of set of sub-agents to reason over than if someone else at Confluent does. Right. How do you persist, you know, the identity information across all these particular endpoints? I mean, in the Glean context, a given request is going to come in and it's obviously, it has identity information associated with it, right?

23:26So, we can make permission to calls wherever we need it. At the agent level, I guess your question is sort of, okay so i guess if we draw the analog back to documents it's obvious in that you know documents have authors and people interacting with it but now if you're the items that you're sort of searching over are other agents or workflows maybe your question is how do we know like who is associated with them and how the identity gets associated there right like if i'm the user that's interacting with the agent and let's say there's some sort of like ui to this agent the presumably agent needs to know like who i am in that organization so that when it makes a tool call, essentially, my identity information can be factored into that tool call.

24:06So it knows that, okay, well, Sean only has access to this subset of documents within the organization. Yeah. I mean, in that sense, it's no different than how Glean manages identity throughout the whole, you know, we can re-leverage our entire like identity infrastructure and platform. But I think the interesting question is sort of on the modeling front of, you know, if Sean's colleague has created this agent, should that agent be more likely to be relevant to Sean, depending on how closely they work together, where they're working in, these same kind of signals, because we can have that identity metadata and that sort of implicit.

24:37The explicit stuff is a given, right? You only can have access to things you have access to. Certain tools can only execute if you have access to them. It's not just documents, right? So we're able to build on all the same clean infrastructure we have to make that work. Given that you have essentially any agent, multi-agent system, you have a lot of like these internal external dependencies, presumably it's going to be probably not running all on one server and stuff like that. And on top of that, you have sort of these unbounded execution plans that might cycle to happen. In a stochastic model, the heart of this is acting as the brain.