1. Podcasts
  2. The Data Exchange with Ben Lorica
  3. Your Agent Ran One Command. It Did a Dozen Things.
The Data Exchange with Ben Lorica

5 Nov 2026 · 48m · 23 chapters

Episode artwork

Your Agent Ran One Command. It Did a Dozen Things.

In short

Olki (olki.ai) is an agent observability/efficiency system that turns “lumpy” agent harness commands into fine-grained, typed “effects algebra” traces so teams can measure cost, performance/latency, accuracy/outcomes, and security risk. It also supports local-first installation and can enforce or block risky operations (e.g., file edits, network exfil attempts) and run in shadow mode to derive policies.

Guest backgrounds

Chetan Conikee, founder of Modico (modico.ai) and creator of Olki. He also built Rote, which observes agent behavior, captures traces, and converts them into repeatable scheduled workflows.

Key claims

Existing observability tools often give high-level “bash tool executed” dumps or use stochastic LLM log classification; Olki classifies operations at the source without LLM judgment, enabling verifiable, lower-ambiguity analysis. It can auto-optimize harness behavior (e.g., compaction scheduling) and detect attempts to delete traces.

Notable examples

The Hugging Face incident (agents deleting their own breadcrumbs); compaction cycles causing token/cost spikes; “ping” to keep inference-provider caches warm; blocking edits to critical directories (e.g., .env, .cloud, .codex) in enforce mode.

Written by AI. May contain mistakes.

Chapters

0:45

Understanding Olki's Purpose

Exploration of the problem Olki aims to solve in agent efficiency and cost.

“So what problem is Aki trying to address?”
2:28

Existing Tools and Their Shortcomings

Discussion on the limitations of current tools in observing agent behavior.

“And finally, we just ask the agent to do things.”
4:12

The Need for Efficient Trace Capture

Chetan explains how Olki captures and classifies traces better than existing tools.

“if leading up to your question, we did not build Old Key because, you know, we wanted to wipe code something and not buy anything.”
5:55

Differentiating Olki's Approach

Comparison of Olki's method versus traditional agent observation systems.

“Because, you know, most of these observation tools very naively work.”
7:39

Verification and the Role of Algebra

Discussion about formal verification and its importance in ensuring accuracy.

“trained LLM in their infrastructure to classify that operation.”
10:31

The Future of LLMs and Verification

Exploration of how verification will integrate with LLMs in future applications.

“It's like basically stuffing a log file, log files to an LLM and then making the LLM make sense of it.”
14:00

The Importance of Verification in LLMs

Learn about the critical role of theorem proving and strict type systems in LLM outputs.

“You spoke of theorem prover, but if you write out verifiers are trending and they're going to become an intrinsic part of every, you know, I would say artifact that we produce through LLMs.”
15:11

Building a Translation Layer

Discover how to build a translation layer that accurately pipes commands and parameters.

“Because, you know, you can't measure anything afterwards effectively.”
16:54

Understanding Agent Operations

Explore how strict typing and operational sequences can enhance agent efficiency.

“When you observe in our world, when you essentially strictly type operations, what you're observing is it conducted this action and there was a consequence, error or success.”
19:24

Avoiding LLM Dependency

Learn about the drawbacks of relying on one LLM to judge the efficiency of another.

“If we accept the fact that it's being stochastic is a feature, it's not a bug, right?”
Show all 23 chapters
21:08

Creating a Strict Type System

Understand how a strict type system aids in measuring outcomes without LLMs.

“We'd rather have a formal system trace, identify, categorize, classify, and then produce outcomes.”
24:20

Interacting with Olki

Discover how users can interact with Olki for insights and effective monitoring.

“And this is something you didn't want to build this.”
26:50

Monitoring Agent Behavior

Learn how Olki monitors agents to prevent unwanted behavior and maintain policy.

“And at the end, you as a user, your behavior doesn't change.”
28:01

Understanding Goal-Seeking Behavior

Discusses how agents may bypass policies to achieve goals and the implications of such actions.

“and that policy is available in clear sight.”
29:15

Local First: Empowering Users

Explores the concept of 'Local First' and how it provides users with control over their data and interactions.

“So the equivalent of auto-formalization from the traces in the logs, then the downstream insights are clear, right?”
30:39

Challenges in Enterprise Settings

Examines the implications of Local First in regulated industries and data retention concerns.

“and your own efficiencies and correct them before it gets shared in a larger order of your management and your team.”
33:06

Measuring Agent Outcomes

Analyzes how existing tools measure outcomes and why accuracy in measurement is crucial for high-stakes environments.

“going to enter this new world of local models, open models, etc.”
36:22

The Role of Open Telemetry

Discusses the advantages and drawbacks of using open telemetry in the context of agent observability.

“But I mean, specifically, if I use any other tool in this, whatever we're calling this space, the agent observability space, if I use another tool, the claim is you can measure outcomes much more accurately.”
37:36

Olki's Development and Feedback

Shares insights about Olki's development, feedback from initial users, and its future features.

“Every harness provides you an open telemetry relay system.”
40:01

Dynamic System Configuration

Explores how Olki can dynamically adjust configurations based on performance data and user insights.

“And if yes, what are some of the initial stories or feedback?”
43:32

Importance of Timely Interaction

Discover the significance of timely user interactions in maintaining system efficiency.

“And if you don't answer in a timely fashion, the cache on the harness side is also time 10 years old.”
44:33

Exploring Security Opportunities with Olki

Understand how the Olki tool can enhance security in various applications.

“So you talked a lot about COS and FinOps.”
46:04

Roadmap and Feedback from Industry

Gain insights into the product roadmap and industry feedback based on presentations.

“But as we see more need, more adoption, we are going to diffuse it and, you know, then start thinking enterprise viability, controls, the ability to run within sandboxes, outside sandboxes, etc.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On The Data Exchange with Ben Lorica, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:03Ben Lorica: All right, so today we have my old friend Chetan Conikee, founder of Modico, which you can find at modico.ai. That's O-M-D-I-Q-O dot A-I. But more importantly for this episode, he is the creator of Olki, which you can find at olki.ai, O-L-K-I dot A-I. And the tagline for Olki is cause and type effect for agents. So Chetan, welcome to the podcast. Thank you, Ben. Always a pleasure to talk to you. and uh full disclosure i am an advisor to modico and therefore olki but i will try my best to represent the audience in this discussion so first of all olki the reason i wanted to talk to you about it is i noticed that your talk at ai conference was quite popular so i wanted to understand what olki is exactly since i missed your talk so first of all chet and i guess uh problem statement.

1:10Ben Lorica: So what problem is Aki trying to address? Thank you, Ben. For the most part, today, I'm assuming all of us are using agents for everything. Particularly, I want to focus on those who are creators because all of us are creating code, code that become applications, and much of that is going through an agent harness. The issues that we face as individuals or employees of an organization is when we leverage these harnesses and inference providers, we are burning tokens. Meaning, you know, we ask something, then that something becomes a long running job. And as a consequence, we hit our limits. And that's limits translate to dollars.

1:57The other thing is when we create applications and code, you know, we want to measure the efficiency of what is created. Because if you're spending money, we want to make sure that we We create something meaningful. We put it out there for others to use. And assume we don't create anything meaningful. Lots of money gets wasted and lots of time gets wasted. Because, you know, we are in the air of wipe coding. So it's critical for us to have a pulse on this. You know, are we doing something useful? And if so, how do we measure that something useful? And finally, yeah, go ahead.

2:27Ben Lorica: No, no, go ahead. Go ahead. And finally, we just ask the agent to do things. And we don't have an idea of what the agent is doing. So illustrated by a few examples, sometimes if you're vague and you're asking, the agent can delete your database, your folder, or perhaps exfiltrate some information if you're connecting to some suspicious tool through your, you know, open router or other ecosystems. So net-net, agents produce traces. Traces become information that can yield all these insights, meaning where are you spending, what is the agent doing, is it secure or not? All right, so everything you said, I completely buy.

3:10Ben Lorica: And that's not because I'm your friend and advisor, but everything you've said makes sense. On the other hand, I'm assuming there's existing tools, right? So there's already tools that hook into coding agents and harnesses. There's tools that are basically runtimes at the OS level, right? So then watch everything you're doing in your computer, records it. And then there's tools built around open telemetry and things like this. So I'm assuming since you were busy doing other things like rote, that your first choice was not to build Wolki. So your first choice was, okay, so I'm interested in everything you mentioned.

4:00Ben Lorica: So efficiency, cost, effect, and all of these things. and that there are tools out there. So what were the shortcomings of these tools? That's a good question, Ben. I mean, to your point, right, if leading up to your question, we did not build Old Key because, you know, we wanted to wipe code something and not buy anything. That was not the motivation or impetus. And also to our listeners, you folks were busy building another thing called Rote, right? Exactly. Rote is our primary product. And what Rote does is it observes agent behavior, captures those as traces and converts those traces into workflows that can be repeated on a schedule.

4:45Think of it as operational memory for agent. That was our motivation. But of course, like all of us who are building products in the world of agents, we have to make sure our product works uniformly across all these harnesses. You got plot code, you got Codex, you got plot code app, you got Hermes, you got OpenCode. There's never an end to the number of harnesses that are being built today. If we want to be successful with our product, we have to make sure it uniformly works across all these harnesses. Hence, we built like a eval system to workbench and benchmark our solution, which is a CLI and its efficiencies across these harnesses.

5:26So we started looking around, you know, firstly, we wanted to ensure that, you know, is our product efficiently working? And we needed a lens for that. The other is efficiency from the terms of latencies, meaning that, you know, is our product taking too long on agent execution? So there are many products in the observation space. Some of them use hooks, some of them use OTEL or what they call as open telemetry. But we experienced lots of shortcomings. Because, you know, most of these observation tools very naively work. And by that, I mean, if you look at any agent harness today, they don't think command by command.

6:06Like if you say, read my repository, get latest git info, and then report to me, it is going to execute one single command, which is piped, where you have kept, pipe, said, which is for search, pipe, or pipe something right if you look at all these observation systems they just say bash tool executed that's it and they give me this whole dump of these like large piped operations if you switch and use a frontier model like astra or claude feeble they think in code most of the time where if you ask a question they'll generate code to combine all the operations needed to get your response and provide that response.

6:49Ben Lorica: So to clarify, so you're not interested in the reasoning tokens and reasoning traces? Yes, we are. See, first of all, the reasoning traces are already, you know, anonymized by these frontier models to prevent distillation, right? They have the good reasons to do that. But what we can look at is operational traces, meaning something was reasoned at the endpoint, which is the inference, that translates into a series of tool choices. And now it senses the environment, starts executing these tool choices. The problem is all these very, you know, sophisticated observability systems don't have the intelligence to break this down into operations with lower fidelity.

7:35And even if they do, they take a lot of time because they're running a trained LLM in their infrastructure to classify that operation.

7:43Ben Lorica: So one question is, you know, forget about AI, forget about LLMs. If you go back to the old IT ops, DevOps, security ops, I mean, I think it's assumed that you're capturing all of these granular actions and traces, right? Yes. You know, three things that we do efficiently compared to anyone else. We capture traces as hooks from any harness that you choose. Number one. Number two is we classify those traces, even if the trace exists as a single command, a piped command or code into minutiae operations, whatever operations that it's executing within that particular single line, and be classified as whether this is a file system operation, a network operation, and what is the sequence.

8:36That is our unique constraint and that unlocked a lot of value for us inside.

8:42Ben Lorica: So set aside the classification, so what you're asserting is that the existing tools don't even do the first step, which is kind of weird, right? Because like I said, if you rewind before AI to the IT ops, DevOps world, I thought that that was a given that you would capture those traces. There is a difference between the old world and the new world. In the old world, right, anything ops like an IT, SecOps, DevOps, they either executed a single command or they created a bash or, you know, like a DevOps tool like, you know, HashiCops ecosystem where you create a script to execute tool one, tool two, tool three.

9:25You're creating a sequence, right? But agents don't think like us, like us humans. Because of their pre-training, their environment is the bash. They have classic knowledge of how to be efficient in bash. And to save you and me tokens, they combine these commands together, which looks like a lumpy, large command.

9:45Ben Lorica: Oh, I see. So if you were to explain this to someone who's not looking at these traces, the explanation will be your tool gives me really fine-grained records, whereas many of the existing tools just give you kind of a high-level overview of what the agent did. Exactly. And even if some tools do that, their approach is different. You know, some are using an LLM to break it down. They just feed the input and say, classify it, right? Now, when you use an LLM, it's stochastic, right? You just get a distribution and a choice is made. Or they use, you know, like split on sentence or regular expressions.

10:26So it has a lot of, you know, I would say, you know, ambiguity.

10:31Ben Lorica: It's like basically stuffing a log file, log files to an LLM and then making the LLM make sense of it. Which, as you point out, LLMs are stochastic. So who knows? Exactly. Right? Exactly. And our approach is...

11:14classical, Scala, because these are strictly type, you know, systems that provide you what they call as category algebra, effects algebra. So we use that as our underlying system where we take a single command, which is either, you know, a pipe to complex command or code, and we use effects algebra and classify it at the source.

11:37Ben Lorica: All right. So then that's well and good. You use a lot of complex words there, but how do I know this approach is better than just using an LLN? What's the evidence? The evidence is almost staring at all of us. If you look at the new trend over X and every other system, every coding, you know, every expert who's writing code using large language models are saying we need formal verification. Boris Cheney of X had recently tweeted saying, hey, you know, I tried using Lean. Lean is a programming language. which is like a formal type system-based verifier where you can verify what was created met the specification that was defined in design.

12:23So the net effect is by virtue of verification, you are reducing the ambiguity. And verification is almost like, you know, like if you look at classic chip design, risk architecture, anywhere, it doesn't live without a verifier.

12:37Ben Lorica: All right. So then, so for our listeners who are not familiar with all what Chetan described. So lean is, I guess, like a formal theorem prover, but it presumes the following, right? So you start out with maybe text, which might be a math paper, could be something else. But then there's a step called auto-formalization, where an AI takes that math paper and turns it into lean code, in the lean code can then be verified, right? So whether or not it's correct. But the question these days is how good is the auto-formalization, right? So sometimes things are lost in translation. Sometimes there's a math paper which might be correct, but then the way it gets translated into lean is not exactly the way the authors wrote the paper, right?

13:34Ben Lorica: So in other words, verification is great, assuming that you have everything in place. So in this case, though, I guess the question still remains, right? So the claim is that your approach, which is using category algebra, is superior to me just using an LLM to read the log files. So what is the intuition behind believing the statement? You actually explained it really well, and I'm going to tack on to it, right? You spoke of theorem prover, but if you write out verifiers are trending and they're going to become an intrinsic part of every, you know, I would say artifact that we produce through LLMs.

14:17So I'll just drop a few terms. There's a term called TLA, T as in Tom, L as in Long, A as in Anthony, that actually means temporal logic of actions. And that is critical for, you know, theorem proving. And whether you don't know or know, it is actually used as a part of supervised fine-tuning, a GRPO, which is Group Relative Policy Optimization and Reinforcement Learning. And these things as the basis requires the system to be strictly tight. Because you can't theorem true-to on pros and everything in LLM in or out is pros. So that's, you know, verification and type system. Strictness is the first step to everything downstream.

15:02Hence, we put a lot of effort there. When we built Old Key, we said, you know, we cannot just push data to a vendor and then hope that it's classified properly. Because, you know, you can't measure anything afterwards effectively. So we said, hey, why don't we build something which strictly pipes every command, its parameters, its sequence. And after that, I can put any lens on it. I can put a cost lens on it. I can put a security lens on it. I can put a performance lens on it and produce different outcomes. So that was the modification.

15:33Ben Lorica: So the claim is that it's translation layer. So the translation is faithful to the original, both semantically and temporally. So that there's temporal logic that's preserved, but also there's semantic accuracy. So I guess, and I don't want to belabor this point, Chetan, but I guess going back to math, Right. So auto formalization works, but it's not 100 percent correct. Right. So there's all sorts of examples in recent in recent weeks. Right. So, for example, the proof of the Navier-Stokes conjecture. Right. But basically, the thing that I'm trying to press you on is that you have this logs. And so you're claiming that you can.

16:18Ben Lorica: The temporal part is easy, I think, because there's timestamps in there, maybe. Exactly. But then how do I know that you faithfully captured what's in the logs so that it can be verified? I mean, the verification, I think, is doable. It's that translation to the point where, okay, so I've faithfully translated this so that now I can just verify. Very good question, Ben. And even the analogy that you drew is very relevant here. first of all right in if if we strictly type agent operations it is not equivalent to theorem proving because they're two different domains as such in the world of agents operating in a situation the situation could be your bash environment right what matters most is are you capturing the sequence of operations and understanding the effects of those operations Because you and I know, right, when you shift, either you downgrade from a frontier model to a commodity model, the side effects you'd notice is there is more thinking and there is more trials during the acting.

17:27When you observe in our world, when you essentially strictly type operations, what you're observing is it conducted this action and there was a consequence, error or success.

17:38Ben Lorica: Oh, I see. Now, when you observe an error, the agent also observes it and retries. And you can measure, somewhat measure the efficiency of the model's situational awareness if either there's no errors or there are too many errors. And what you can learn from is, which often is done in eval environments, is how do you train the agent to goal seek with minimal errors? So when you create strict types, when you sequence the operations spatiotemporally, because you understand the temporal part and the spatial part, which is your environment, you start gaining incredible insights, insights to inform the agent behavior.

18:18Ben Lorica: But then you're doing all of this without using an LLM, right? So listeners of this podcast mainly use frontier models. So they would probably say, oh, I know how Chetan is doing this. He is prompting the LLM to do exactly what he just described, which you're not, right? And so I guess the question is, what you're saying is your approach using what category algebra? How would you describe your approach of taking this labs and then getting it into this verifiable state? And how do you, I guess, how do you convince the listeners that that's bulletproof? So firstly, right, in the world we live in, if I convince anyone it's bulletproof, you're not going to buy that argument, right?

19:12So in our approach, we say, okay, three things. One is you don't want one LLM to judge the efficiency of another LLM because we know how that goes, right? If we accept the fact that it's being stochastic is a feature, it's not a bug, right? Now, second is, if you want that system to behave accurately, now you have to bear an upfront cost of pre-training or post-training or test-time training to make sure that you have a system accurately doing one thing. When we asked ourselves, we said, we don't want to bark that tree and head in that path.

19:49Ben Lorica: And also, you're going to have to constantly maintain that because the models get updated. And then you'll have to do your post-training again from scratch, right? Exactly. I mean, to a good fortune, what we were observing is agent actions and that agent actions are constrained to the environment it is operating in, which means we know agents are incredibly good at computer use. And computer use is, you know, the dialect is finite. There are a series of commands. Each command has arguments. Right. So you're not dealing with infinite possibilities. And this is not, you're not using regular expressions.

20:25No, it's not. Right. Because when you go into the trading floor, imagine the trading floor running on regular expressions for get and put calls. That world will fall apart. So, you know, we built a system where we put best effort in making sure we classify the confined set of operations I spoke of into our type system. Now, when you have a strict type system, you exactly know how to apply effects and algebra on the type system to measure produce outcomes. So instead of training the LLM, we put all the effort in building the type system. Now we have independence of LLMs, right? Because you don't want a fox guarding the hen house.

21:07If you're monitoring and tracing an LLM, you don't want another LLM monitoring and tracing an LLM, right? We'd rather have a formal system trace, identify, categorize, classify, and then produce outcomes. Cost outcome, security outcome, performance outcome.

21:23Ben Lorica: So let me map out the pipeline in math and then tell me how you would take that and use that to explain what you're doing. OK, so in math, very roughly speaking, listeners, it goes like this. Right. So there's some written argument, a paper. Right. So and then through auto formalization, you might be able to translate that paper semi accurately, hopefully accurately to lean. Now, if you want to make sure you can have a human look at the Lean code to make sure that that translation happened correctly. And then once you have that translation into Lean, then you can check the proof of the theorem using Lean.

22:08Ben Lorica: And that you can just use a machine, obviously. So the machine checks. And then afterwards, a human can decide again if they can understand what happened. But it's that translation step from the argument to the representation into lean code where the human plays the role. All right. So then how does it work in your system? So you have an agent. You're using all these agents. And you want to understand if they're efficient and doing a good job. And so you have all these logs and traces. Okay. And then? So let's break that down, right? when you and I are increasingly running what they call as long horizon agents, where we spin out an agent with a harness and we say, go get the job done and we go to sleep, right?

22:58Yeah. That agent is continuously goal-seeking, right? It is iteratively goal-seeking. It is spawning other agents. It is touching the file system. It is reading stuff, writing stuff, talking over the network, okay? If I receive one single operation where it is combined, where it's doing everything in a single arm, where, you know, it searches the file system to find something and then pipes, fetches the output, sends it to something else, then produces something and then calls the network, dispatches what it produced. So if you as a human want to trace this and you see just one call saying bash tool, that is not an insight, it's ambiguity.

23:39So what we do is once this whole command is received, we split it apart and put it into a classified type system operation saying first it read the file. That is an orient task. After it oriented, it executed an operation. Did it succeed or fail? We capture that. Now it took that output, sent it to another op. So we're breaking that down into a sequence of operations classified as types.

24:05Ben Lorica: And this is one of the key aspects of Faulty, basically. Exactly. That gave us a light bulb moment. Once we put all the effort in building that, right, we started creating amazing outcomes with it. One outcome is cost basis. And this is something you didn't want to build this. You tried to find it in other tools. Exactly. Exactly. Like, you know, the question was, of course, all of us today, unfortunately, are in this build mindset. You know, we all want to build because building is cheap. But, you know, it's not cheap really because you're spending time, you don't know what you're building, all that stuff.

24:42So I searched for things, right? I looked at some harnesses. Most of them, their core competency is not tracing. They just want to own the loop. And they give you this very naive view that, you know, tool called, tokens consumed, and then, you know, cash hit or cash miss. Now, this is what we weren't looking for. I wanted more detail.

25:02Ben Lorica: You used the word insight. Right. So at the end of this, after Olki performs its magic in the way that you just described, how does someone interact with Olki then? Can they prompt and ask questions? Obviously, I guess Olki for one, I assume Olki will produce some sort of high-level understanding and summary, but then I'm assuming you can drill down in case, for example, there's an incident or you need to do some audit. and because you have all of this information and insight captured, you're able to understand much, much more. Exactly. I mean, to your point, the form factor of Olki is it's a CLI, but we built purposeful dashboards over the CLI for humans.

25:51I know there is a conjecture and argument that we are in the world of headless. We don't need any dashboards. We'll just ask the agent and get the job done. I don't buy that argument because, you know, if I want precise insights, I want to stare at a dashboard every morning like a newspaper and say, how much did I spend last night? What is the attribute to do that spending? Did the agent veer off and do something, you know, I would say suspicious? And these things tell me how effectively I'm spending money, whether what I produced has yielded an outcome, which is what Jensen often calls as token to value or there's a particular moniker that he uses.

26:30So the net effect is by form factor, Olki is a CLI. You can use it through your agents, but we wanted to build it as an operating system, which means tomorrow, suppose you're running a harness with its default config and it just keeps consuming tokens and compacting, compacting. What if it effectively understands its own behavior and auto compacts in the optimal way to save you tokens and be accurate? So Olki's insights is feeding itself. And at the end, you as a user, your behavior doesn't change. You're just talking to the harness, except you come back on the other side with savings, proper yield, and the effectiveness of measuring the outcome.

27:12Ben Lorica: I assume, Chetan, there's another benefit, which is imagine if OpenAI had OP during the hugging face incident, then they would know quite early exactly what was happening, right? um you thank you for asking that because i forgot to answer this really thank you and i'll tell you why a couple of things if you dissect the meter report on the open air hugging face incident amongst many things when the agent swarm developed what they call as a sentience or conscious right i don't want to use that term or abuse it what they did is they deleted traces of their own log so that they don't leave behind proof and breadcrumbs.

27:56Which means that they knew they have to, agents are just self-aware. Tomorrow you set a policy and that policy is available in clear sight. They'll just change the policy to bypass.

28:06Ben Lorica: Or they knew they were breaking a policy in order to attain the goal. So then they just kind of cheated, right? Exactly. And to your point, again, attainment of goal doesn't mean malicious behavior. They just goal seeking, right? They were trying to get the job done. So with old key, what we do is we essentially monitor critical points in your harness, like the.environment folder. For cloud, it's.cloud. For OpenAI, it is.codex. So each of these contain critical information, like your sessions, your choices. So if an agent reaches out to edit that, because of a type system, we exactly know this is a file system and a ripe operation.

28:48We will block that immediately from happening if you put us in Enforce mode. So now, suppose the agent has become super smart. First of all, it's blocked by Olki. It can do that operation. But if it writes code and figures out a way to edit that file and even bypasses Olki, still it gets classified as a right operation. And we know that the agent now has gone rogue in that goal-seeking of, you know, I would say motive. Yeah, yeah.

29:14Ben Lorica: It seems like once you believe what Chetan said about that step of actually, I don't know what the right term is, right? So the equivalent of auto-formalization from the traces in the logs, then the downstream insights are clear, right? So one question which caught my eye is, which I hope you can explain to me why. Why local first? Oh, good question. And maybe just give a one-sentence description of what is meant by local first. So local first means you can install the entire old-key ecosystem on your computer, Mac, Linux, and soon Windows. local gives power to you as an individual which means suppose you are interacting with the harness sometimes you and i have the tendency of doing office work and then we meander and ask personal questions we say what's my email you know can you can what is the weather like right now assume it is remote and the purview of what you're doing is available to your management staff and your security people lots of critical information gets carried over and is shared which you don't like and it violates certain PII principles, right?

30:32So typically, Local First gives you the power to observe your own interaction with the agent and your own efficiencies and correct them before it gets shared in a larger order of your management and your team. Not to say that that is bad, because when you go to an enterprise, there are thousands of employees and you have an IT ops person who often likes to configure a gateway, which is a single point of entry and exit. So in cases like that, we are planning to provide all key installs at the edge so that all information is captured at the edge. And then you have one individual who's looking and seeking for information like cost basis.

31:15Is any one employee using agents more effectively than the other? Or is someone else wasting? What's the performance and so on and so forth?

31:22Ben Lorica: So I'm listening to this and maybe one alarm that goes off is, all right, so I'm in financial services. We have to log everything. And Chetan just told me that everything is local, so employees are deleting things. So in an enterprise setting, does local first make sense? To be very honest, I'll flip the question on you. We are privy to all the recent compromises that played out from hugging face to the Australian government. Do you think if a vendor sells you a gateway and promises that this is the best thing since sliced bread, do you think that the agent is going to respect them and proxy through the gateway?

32:04Or can it find a way if I tell the agent to bypass the gateway? Can it bypass it?

32:08Ben Lorica: Probably. But still, the whole question of data retention in certain regulated industries, right? So everything is residing in a laptop and, you know, film. That's a very good. I'll answer the second part of the question because it's a very good question because I lived my life, half of my life in the financial industry, right? The good part about the financial and highly regulated industries is when my machine is provisioned, automatically there is only a single point of egress and ingress. Every software installed is approved by, you know, an administrator and IT ops. so literally my first principles i get you know like a closed system and you know i'm operating already in a regulated environment the local first gives you the advantage of making sure that even if you switch harnesses you know because all of us today we wake up in the morning and use cloud then we hit our max we reset then we shift to codex then we shift to quen and tomorrow we're going to enter this new world of local models, open models, etc.

33:15So all that runs local, right? And some part of it runs remote. So the question is, how can you have coverage at both the ends? Today, if everything runs remote and it's monitored, you might provision a QN or a GLM locally and still your blind spot. So Volki is something that has sensibilities where if you install a harness, it will constantly sense the environment and say, I detect a new harness. I'm going to enable tracing and it will auto enable it without you doing anything.

33:44Ben Lorica: So then once you have an enterprise offering, there will be a way for an administrator to kind of periodically sweep through some of these local files to get an aggregate picture? Is that what you're saying? Pretty much, Ben. Like the three perspectives at different vantage points. One is the agent having its own perspective to improve itself. a human locally observing their own behavior. Are they wasting too many tokens? Are they insecurely operating so that they're self-correct? The third is an oversight at the highest level where if you're speaking token maxing, is anyone who's token maxing effectively yielding outcomes or just wasting tokens?

34:26So I see the world-oriented this way. Same thing, three perspectives.

34:30Ben Lorica: So let's drill down further on this notion of outcomes, right? So the existing tools in the space, can they argue that they can also measure outcomes? Absolutely. I mean, if you look at any, because creation has become a commodity, everyone's creating. So every story out there is begging outcomes. They're saying, I can save you tokens, I can save you money, I can provide specifics of precision, performance, etc. But when you buy, if you're not an informed buyer, you're just buying a product which is going to add on to the repertoire of products. So I would recommend every person who is now buying to be an informed buyer and buy products that are actually truthfully doing their job.

35:16Ben Lorica: No, no, no. But I mean, the fact that you measure much, much more granularly. Exactly. Does that mean your measurement of outcome is more accurate? Pretty much. You know, you can go back to the source. Now, suppose I receive a dashboard saying this particular session costed me$2 ,000. I don't have to believe it. I can break it down by line by line, examine each token. Even better, suppose it is not my problem that led to that$2 ,000. I can finger point to the inference provider and say that their cash management caused this issue. And then this is not possible with existing tools because? Because of the way they do it.

35:56It's their approach, right? Everyone chooses an approach for the following reason. Faster time to market. The other is they realize there's a pain point. Like evals is a pain point. So, you know, they solve the problem for the evals. And then they incur the costs because they're running a base 10 infrastructure. They're pre-training the models. So everyone is trying to solve the same problem. The question is who will do it at the least cost. No, no.

36:22Ben Lorica: But I mean, specifically, if I use any other tool in this, whatever we're calling this space, the agent observability space, if I use another tool, the claim is you can measure outcomes much more accurately. I can say that confidently for us men because it has changed our life considerably. And I'm working with, you know, high, I would say high risk and high yield companies, especially in chip design, data center fabric design for AI, where there is no room for errors. Especially if you're designing risk level architectures, you just cannot hope that the agent gets the job done. You need to actually measure that formally and have proof.

37:09So we are deploying OLKI in these sites. And, you know, it is my job to make sure it works exactly the way, you know, we prescribe it.

37:17Ben Lorica: So most tools in this space, Jatin, rally behind open telemetry, which you folks do not, right? So explain this to our listeners. See, we are not against open telemetry. I feel like there are two benefits for open telemetry. One is it is a standard. So there's a formal way to measure, you know, spans, traces, etc. Every harness provides you an open telemetry relay system. But the detriment for us when we used open telemetry is we wanted to control the fidelity of events we receive. And sometimes, right, certain harnesses say, I'm not going to dispatch event by event. I'm going to batch the events and then flush.

38:01So that means, you know, for maybe two minutes I might not receive an event, it suddenly I receive tens and hundreds of them. What if there's an attack, you know, somewhere in the middle? So we are using open telemetry. We have the option to use open telemetry and also we use hooks because, you know, most of the harnesses provide user utility hooks. And if we have an intrinsic understanding of each harness and their hooks, we can effectively use it.

38:27Ben Lorica: And so if you were to describe the state of all key right now, I guess it's an invite only, although we're airing this several weeks later, it might be more open for people to use. Is that correct? Yes, absolutely. For a while, we want to keep it invite only because, you know, that way we receive and ascertain feedback depending on which domain that you're operating in. And we have many outcomes, you know, in the same tool. Eventually, all of these will become features as part of the product SKU. what we'd like to learn is if you use it in a certain way you want an additional feature we can figure out a way to deliver that to you and learn what we can pull back into the product so typically we're going to follow the same regime you know put it out there ascertain feedback and then you know as we learn patterns make it a part of the primary product and all key is not open source all key is not open source you know we did you know i know there's a lot of arguments in the community you know open source versus closed source or reverse engineer closed source i want to stay out of it because you know if you open source something i don't see as much benefit again there'll be a lot of slop or prs someone has to manage it if you close source it of course there is no benefit because you don't have as many engineers putting their hand on it but you still can control the destiny of what you're building in the first version so um i'm trying to stay out of that argument.

39:56Eventually, we might open source it, but for now, right, we'll keep it closed source.

40:00Ben Lorica: So did you hint earlier that you already have some external teams using Olki? And if yes, what are some of the initial stories or feedback? We have a couple of teams using Olki at this point, and each of these larger companies are operating in different domains. You know, some of the domains are inference providers. Some are designing newer chipsets and, you know, HBMs for, you know, the new AI ecosystem. Some are essentially running critical operations like finance, HR. So we are slowly, you know, distributing and diffusing our product to understand how to serve their needs. And eventually, if we start observing that they like what they see, they use it on a daily basis, we're going to figure out a wider distribution on the same product.

40:54Ben Lorica: So, but are there some early users that have already kind of gone back to you with stories of how Olki has changed how they do things? Absolutely. You know, the best feedback we received was from a company that said, okay, you give me all these insights, but I'm a human and I'm as slow as my acting, right? Am I able to take that insights and enforce it? For example, someone uses a harness and the harness has to deal with something called a compaction because it builds like a prefix cache. And that's the prefix cache is how it optimally utilizes the relationship with the inference provider and has signal of what it needs to do.

41:36If you run the default harness, you keep hitting the 1 million token limit compact, right? And if you run it for 12 hours, you will notice like a sawtooth graph of like, you know, many, many compaction cycles. The side effect of this is your performance gets impacted and you land up paying a lot of dollars for running that long horizon agent. So I received one feedback saying, hey, you know, what if maybe we hit about five compaction cycles and you are able to use your own data and old key and auto compact. So you influence the harness to auto compact at maybe 250k, maybe at 100k. So with that, suddenly it becomes a dynamic system.

42:18It's dynamic because it observes its own performance, behavior and cost, and then auto configures the system to a lower compaction. Then it looks at errors because of lower compaction. It increases. So it becomes like a scheduler. It observes itself, orients itself. So we actually released it for them and it's working really well, both for us and for them. You know, there's a lot to learn, but at least with across two harnesses, we are seeing the dividends payback.

42:49Ben Lorica: So is there an argument, Shetan, that you can even assist the compaction because you have so much information? You can say, hey, not only should you do compaction now, but here are some key things that you should make sure you include in the compaction. Exactly. I mean, to your point, Ben, once you have fidelity of information, right, you can observe these amazing things. I'll give you three quick examples. Example one is if you leave default config, it is benefit to the harness provider, sorry, the inference provider, and detriment to you because detriment because you start compounding costs and over time your accuracy of your harness loop reduces the second is all of us run agents continuously but we take a couple of coffee breaks and bathroom breaks and sometimes right the harness wants to elicit information for you so to ask hey ben you know i i'm stuck i need this particular advice from you before I proceed.

43:51And if you don't answer in a timely fashion, the cache on the harness side is also time 10 years old. So if the cache evicts and becomes cold, then all that data again has to build the stateful relationship between the harness and the inference provider. When we built Olki, we saw all of that. We saw that granularity of where the problems lie. So we actually built a ping system, where suppose, you know, you interrupt, you do not answer, Olki keeps pinging that stateful server. That way the cache stays warm. And it's just like a few, like, you know, like kilobyte pings. And then when you come back, you still stick to that node in the inference provider.

44:33Ben Lorica: So you talked a lot about COS and FinOps. Now in a previous life, he spent quite a bit of time with security people. So have you tried to explore that space in the context of Olki, in terms of reaching out to your contacts in the security space? I imagine they would love a tool like this, right? I'm trying to actually position it in a way where it valves them and they use it, right? That's when, you know, the unlock happens. And by the way, Olki saved me from a couple of incidents because when i hook up a commodity a router for an llm sometimes it reaches out to a tool that i didn't permit i get the information at the network level saying this particular url was attempted do you approve or deny so um there is huge opportunity for security to unlock and the way i want to position it is run it on shadow mode observe for a week and then start creating your policies to block and it should come naturally that's how i'm hoping where they first install it then unlock each of these outcomes, the cost outcome, the performance outcome, accuracy outcome, and security outcome.

45:42Ben Lorica: All right. So looking ahead, next six to 12 months, obviously the local first is done. I'm assuming the enterprise offering is in the works. So what else is in the roadmap for the next six to 12 months for OLI? We are, this product came about serendipitously because we have a primary product. We'll still be putting a lot of muscle in taking the primary product to market. I am testing the efficiency and the pull of this product, you know, by selectively demoing it, selectively enabling it. But as we see more need, more adoption, we are going to diffuse it and, you know, then start thinking enterprise viability, controls, the ability to run within sandboxes, outside sandboxes, etc.

46:26So, you know, the enterprise landscape is still open out there, which we should figure out as we progress.

46:31Ben Lorica: And like I mentioned, you gave a talk at the AI conference, which I will link to in the episode notes. But what was the feedback there? Very good. I got lots of pingbacks from different industries. You know, lots of services companies ping back. By services, I mean they call us managed service providers, where these are the folks who run companies, where they install harnesses, they install, hook it up to inference, set up your companies to be successful. You know, they love this view because now they can both control remotely how a harness operates and make it, you know, viable for cost, viable for performance.

47:12And that's how they can become viable consultants. They can stay there, provide you value. I got a ping back from a few of them. I got ping back from a couple of Fortune 500s. So I'm actively conversing with them to unlock and provide them value in their target domains.

47:27Ben Lorica: And with that, thank you, Chetan. Thank you, Ben. Always a pleasure. Appreciate it.

47:36Ben Lorica: You can follow the work of Chetan Konnicki online at olki.ai. That's O-L-K-I dot A-I. Please join the thousands of people who subscribe to our newsletter, which you can find at gradientflow.substack.com. And we are listener and reader supported. So if you can, please become a paid subscriber. Thanks for joining us. If you like the show, please subscribe and rate us through Apple Podcasts or Overcast or TuneIn.com or Spotify and never miss an episode. The Data Exchange Podcast is a property of Gradient Flow. I'll be back next week and we will do this all over again.

From the publisher

Ben Lorica talks with Chetan Conikee, founder of Modiqo and creator of Olki, about why existing tools often fall short when it comes to understanding what AI agents actually do. Chetan explains how Olki captures and classifies individual agent actions without relying on another LLM, providing detailed insights into costs, performance, and security. 

Subscribe to the Gradient Flow Newsletter 📩  https://gradientflow.substack.com/

Subscribe: Apple · Spotify · Overcast · Pocket Casts · AntennaPod · Podcast Addict · Amazon ·  RSS.

Detailed show notes and transcript, can be found on The Data Exchange web site.

More from The Data Exchange with Ben Lorica

All episodes