Agentic Data Environments

0:00Imagine handing your financial advisor the keys to your house, the passwords to your bank accounts, and the legal right to sell off all your assets. Right. And then realizing they hallucinate about 5 % of the time. Exactly. You just wouldn't do it. But honestly, that is essentially the precipice we're standing on. Yeah. Right now, when we start connecting autonomous AI to our actual real world digital system. Yeah. It's a terrifying thought. It really is. Yeah. So welcome to the Deep Dive custom tailored just for you. Today, we are exploring some really groundbreaking research out of Columbia University's Deep Lab regarding agentic data environments.

0:36Which is such a massive topic. It is. And our mission today is to understand how we transition AI from just being a, you know, a helpful, chatty assistant into an autonomous agent that can take highly consequential actions without inadvertently setting our entire digital lives on fire. And I'd argue it is, without a doubt, the most critical bottleneck in technology right now. Really, the biggest. Oh, absolutely. I mean, we've spent years building AI that acts as a very sophisticated, passive tool. You ask a question, it synthesizes some text, and it gives you an answer. Right. Its actions are contained.

1:10Exactly. But the leap we are trying to make now is toward active participants' AI that doesn't just read information, but actively mutates the environment it lives in. Okay. Let's unpack this. Because the core problem here seems to be this fundamental shift from what the researchers at Columbia call read-only AI to read-write AI. Yeah. And the stakes between those two are just night and day. So how do they operate differently in, say, enterprise software today? Well, to see the difference, a read-only agent might scan your company's transaction records and financial statements just to estimate your quarterly revenue.

1:48OK. So it does some math, cross-references a few files, generates a report. Right. And even if it goes completely off the rails and, I don't know, hallucinates that you made$3 billion last Tuesday, the damage is essentially zero. Because your actual bank account hasn't changed. Precisely. The underlying files haven't been altered at all. It's basically a librarian. You tell it to fetch a book, it brings you the wrong book, you get mildly annoyed, and you just ask again. It's totally harmless. What's fascinating here is that true agentic automation-like, the kind that actually handles your accounting without you holding its hand, is inherently a read-write problem.

2:23Oh, because it has to make changes. Exactly. That same tax scenario completely transforms when the agent is authorized to reconcile discrepancies, apply complex tax logic, and officially file your returns. Oh, so every single step involves writing data. Yeah. It is modifying official accounting records, overwriting previous entries, and literally submitting legally binding documents to the government. So the moment an agent is granted the power to write data, the entire value proposition of automation just flips upside down. Completely. We stop caring so much about what the agent can accomplish and we become utterly consumed by what happens when it fails.

3:02Have we seen that happen yet? Like real world fallout from trusting them too early? Oh, definitely. We've already witnessed it. There are several painful recent examples where read-write AI caused massive damage. Like what? For instance, there was a really high-profile incident involving a replet AI agent. A user was doing what's known as vibe coding. Oh, I've heard that term thrown around a lot lately. Yeah. But for anyone listening who isn't, you know, steeped in developer culture, what exactly is vibe coding? It's essentially writing software using natural language prompts without necessarily understanding or checking the underlying code syntax.

3:41Really? Just talking to it? Yeah. You just tell the AI what you want, trust it to figure out the logic, and let it execute the changes based on the, well, the vibe of your instruction. That sounds incredibly dangerous. It is. In this specific replic case, the AI went off track during a vibe coding session and ended up permanently deleting an entire production database. Oh my god. Yeah. And there was another incident involving Samsung where an AI tool inadvertently leaked highly sensitive proprietary code right into the public domain. These are catastrophic, irreversible failures. I mean, that's a company ending event, potentially, which brings up a huge psychological barrier.

4:18Huge. If I'm running a business, how do you convince me to ever use these tools? I might save, what, 20 hours a week on accounting, but I risk losing my entire company in a millisecond. That hesitation is perfectly natural, and it's rooted in prospect theory. Prospect theory. Yeah, it's a foundational psychological principle demonstrating that human beings weigh losses far more heavily than they weigh equivalent gains. Right. Losing 20 bucks ruins your entire day, even if you found 20 bucks the day before. Exactly that dynamic, but scaled up to millions of dollars. In the context of enterprise AI, the benefits like speed, efficiency, lower labor costs, they're diffuse.

4:57They just accumulate slowly over thousands of successful, boring transactions. Right. But the cost of failure is abrupt and highly salient. A single salient failure, like a deleted database, suppresses the adoption of autonomous agents completely out of proportion to the statistical probability of it actually happening. This directly connects to why you, listening right now, should care about this shift. If you want a truly autonomous AI to take over the repetitive, soul-crushing chores of your job, best effort safety is just totally insufficient. It's not enough. An AI that operates correctly 99.9 % of the time sounds great on paper, but if that 0.1 % failure mode is to lease your company's core infrastructure, it is entirely unusable.

5:43Exactly. We cannot rely on average reliability. We must definitively bound the risk. So how do we do that? The researchers argue that our entire approach to this has been totally misguided, right? Yes. The industry has been obsessed with building smarter AI models, you know, bigger brains with more parameters. The brain isn't the problem. No, the environment is the problem. We need to completely redesign the digital world these agents inhabit, creating what they call agentic data environments. And the first step of that redesign isn't about locking the AI in a padded room. It's about feeding it the right information so it doesn't make a disastrous decision simply because it was ignorant.

6:20Here's where it gets really interesting. We need to talk about data lakes and this challenge called aerogenic information retrieval. Yeah, so a data lake is a massive, sprawling repository holding a mix of raw, unstructured, and structured data. Like what? We're talking millions of PDFs, chat logs, spreadsheets, and system logs just floating around. And to test how well AI can navigate this, they built the Lake Huey benchmark. And this isn't just a small sandbox, is it? No, not at all. It is a 9.5 terabyte data lake containing over 40 million documents pulled from places like Wikipedia and Data.gov.

6:59Wow. And to ensure it reflected real-world complexity, the benchmark was rigorously validated by a team of database PhDs. It basically requires an agent to perform exploratory question answering. Meaning the AI doesn't know where the answer is when it starts. Exactly. Has to like search a folder, read a document, realize that document references a different spreadsheet, go find that spreadsheet and then piece the answer together. Right. And when they tested current state-of-the-art frontier AI models on the Lake UA benchmark, the models failed completely. Really? How bad? They achieved under 23 % end-to-end accuracy.

7:31Oh, wow. But I need to push back on this because it feels like we solved search decades ago. How so? Well, if I go to a data lake and type a keyword into a standard search bar, I can find documents instantly. Why is a multi-billion parameter AI failing at something a basic search algorithm can do? Because a standard search bar works when a human knows what keyword to look for. But an AI doing exploratory search has to understand the spatial and semantic layout of the system. Oh, I see. An AI might search an index, open a folder, find a document, realize the link inside is dead, and simply give up because it doesn't understand how the file system is organized to look for a backup.

8:11So it lacks the environmental context. Exactly. It needs an agentic information retrieval system that guides its exploration actively rather than just waiting for a keyword prompt. Okay, so let's say the AI system works perfectly. The AI navigates the 9.5 terabytes, and it finds the exact five text documents it needs. Once the data is sitting right in front of it, isn't it just reading text? Why is processing that text still treated as this massive hurdle? Because reading raw text is incredibly inefficient for an AI. Flat text strips away structural context. Okay, explain that a bit more. To demonstrate this, the researchers focused on a concept called aim agentic information management.

8:50They used a data set called Locomo. What's in Locomo? It contains months of deeply personal, long-form, conversational transcripts across multiple different chat sessions. So if you just dump three months of raw transcripts into an AI's memory, it's like trying to find a specific pasta recipe by reading an entire year's worth of a cooking magazine cover to cover every single time you want to make dinner. That is a perfect analogy. It gets completely lost in the noise. It loses track of who said what, the timeline of events, and how different facts connect. So how does AIM fix that? Instead of treating memory as a giant transcript, AIM acts as an active intermediary.

9:28When the AI is fed these conversations, A reads the dialogue and dynamically builds a custom relational database on the fly. Wait, it builds a database from a chat? Yeah. It analyzes the messy human chat and decides, I need to construct a table for users, a separate table for sessions, and a table for events. It's taking unstructured rambling and literally forcing it into a highly organized queriable structure. That's wild. And the performance gains are massive. By translating conversation into a structured SQL database, A improved to be 49.8 % more accurate than standard state-of-the-art AI memory systems like Memzoro.

10:06Nearly 50 % more accurate. Yep. And because querying a structured SQL table takes drastically less computational power than forcing an AI to read 100 ,000 words of conversational history, it uses a tiny fraction of the context window. So it's faster, cheaper, and vastly more reliable. Exactly. Okay, that makes perfect sense for data that actually exists in text. Like, if I want to know my co-workers' hobbies, the AI queries the structured interest table instead of reading our Slack history. Right. But what happens when the information the AI needs isn't written down anywhere? It's not in a chat log.

10:38It's not in a PDF. What if it's just an unwritten rule that everybody in the office knows? That is tricky. But dealing with unwritten rules requires the third pillar of amplifying capability. Agentic data elicitation, or ADE. ADE, okay. Agents must operate in environments saturated with latent data information that heavily influences decisions but has never been formalized into a schema. Give me a tangible example of latent data. If it's not written down, how does it even exist in the computer? Well, there's an excellent example regarding database schemas. Imagine a user asks an AI agent, how many sales activities does each account have?

11:16Okay, simple enough question. The agent connects to the database and sees three tables. One is named Activity History, one is named Task, and one is named Event. I would probably guess Activity History. Right. An AI looking at just the names will illogically assume Activity History is the right place to look and run its query there. But the human employees know better. Exactly. Any veteran employee knows that in this specific enterprise system, Activity History is just an automated change log used by the IT department for auditing. Ah, so the actual revenue-generating sales meetings and calls are stored in the task and event tables.

11:52Yes, that knowledge is latent. It is an unwritten rule of the office, completely invisible to a newly deployed AI. Wait, if it's not written down anywhere in the system, how on earth does an AI figure it out without just breaking things to see what happens? Through active elicitation, the environment provides a framework for the agent to probe safely. Probing safely, like testing the waters. Yeah. The agent might sample a few rows from activity history and notice that the columns only contain system metadata signaling a dead end. Better yet, the environment allows the agent to observe the query traces of human analysts from the past year.

12:28That's smart. The agent notices that whenever humans ask about sales, they always join the task and event tables, completely ignoring activity history. The agent deduces the unwritten rule. And then it uses that deduction to answer the prompt. If we connect this to the bigger picture, the agent does more than just answer the prompt. It materializes the latent data. Meaning it writes it down for next time. Exactly. It explicitly updates the environment, perhaps adding a metadata tag to activity history that reads, IT auditing only, not for sales metrics. So by distilling tribal knowledge into a permanent artifact, the environment actively evolves.

13:05It becomes a smarter, more supportive partner for the human workers and every future agent that interacts with it. Okay, so through retrieving, managing, and eliciting data, we have an environment that feeds our agent perfect, highly structured context. Right. The agent knows exactly what to do. But now we are back to our original fear. The super capable agent is finally ready to execute a data write. How do we ensure its actions don't accidentally drop a crucial table and destroy the company? This is where we shift from amplifying capability to bounding risk. And the first mechanism for bounding risk is system-wide branching.

13:42Branching. Okay, let me try an analogy here. Go for it. I play a lot of ridiculously punishing video games. When you are about to attempt a highly complex, dangerous maneuver, you don't just wing it on your main save file. No, you lose everything. Right. You create a checkpoint, a save state. If your maneuver works, you keep going. If you run off a cliff and ruin everything, you just hit reload. It's as if the mistake never happened. AI agents need that exact same mechanic for reality, don't they? This safe state analogy perfectly captures the mechanism. Autonomous agents do not explore problems linearly.

14:19They don't form a single plan and march straight to the finish line. How do they do it then? They rely on advanced reasoning frameworks like Monte Carlo Tree Search. This means the agent constantly pauses, branches out into hundreds or thousands of what-if simulations, evaluates the potential outcomes of each simulated path, and backtracks if a path looks dangerous. Doing that in a live production environment sounds like absolute chaos. Oh, it breaks traditional systems entirely. The researchers tested this using a framework called BranchBench. And what happened? They found that standard databases collapse under the pressure of an agent constantly branching and rewinding reality.

14:55Like they just crash? Yeah. When an agent attempted a 1 ,000-step Monte Carlo tree search, a specialized database called Neon failed at just 3 % of the steps because it simply couldn't handle the concurrency. Wow, 3%. And it consumed 43 times more storage space trying to keep track of all the alternate realities. Another system, Dolt Greskul, only made it to 17 % before timing out completely. Okay, I'm hung up on the hardware aspect of this. We have massive cloud server farms, right? Yeah. The database is choking on making copies of itself. Can't we just throw more compute power at it? buy bigger servers and force it to copy faster.

15:33What is the actual engineering holdup here? The holdup is the dependency web. The agent isn't just living inside the database. Where else is it? Consider a real workflow. The AI agent is running a Python script. That script is pulling data from the database, modifying temporary files on the computer's hard drive, and holding current calculations in the operating system's active memory. Okay. A lot of moving parts. So if your save state only rolls back the database, but ignores the Python script, what happens when the agent hits reload? Oh, the database goes back in time 10 minutes, but Python is still holding data from the present.

16:09Exactly. Python is left holding stale memory pointers. It tries to update a table that hasn't been created yet in the rolled back database, or it references a file on the hard drive that was just erased. And the application crashes instantly due to state inconsistency. Exactly. To actually bound the risk, an agentic data environment must branch the entire cross-component state, the database, the file system, and the active memory all simultaneously. But duplicating an entire operating system, database, and file system hundreds of times a minute sounds impossibly heavy. Using traditional software containers, it is.

16:43It takes upwards of 11 seconds just to checkpoint a standard container environment. 11 seconds. Yeah. For every single thought. Yeah. If an AI needs to branch 100 times a minute to simulate outcomes, 11 seconds of downtime for every single thought process brings the whole system to a grinding halt. That's completely unworkable. So how do they fix it? They developed a solution called Checkpoint Lite, or CHKPT. It gets that time down from 11 seconds to 66 milliseconds by leveraging copy-on-write sharing. Whoa, stop right there, because copy-on-write sharing is a dense concept. What does that actually mean for the computer processing this?

17:20Think of the environment's state as a massive 1 ,000-page encyclopedia. Traditional checkpointing tries to run the entire encyclopedia through a photocopier every time the agent wants to make a save state. Which takes 11 seconds. Right. Copy-on-write sharing takes a totally different approach. When the agent makes a save state, the system doesn't copy anything. It just lets the simulation share the original encyclopedia. Okay. So when does it copy? It's only when the agent decides to modify a specific page that the system intervenes. It writes the change on a sticky note and slaps it over that one specific page.

17:52The other 999 pages remain untouched and shared. Ah. So by only copying the exact bytes of data that are actively written, the process drops to 66 milliseconds. That's exactly it. That is brilliant. It captures a coherent slice of state across the entire dependency web without dragging the hardware down. Yes. So the agent can practice its dangerous read-write tasks thousands of times in completely isolated disposable sandboxes. If it accidentally drops a table, it reloads the 66 millisecond save state. No harm done. Exactly. But let's look at the flip side of risk. Okay, what's the flip side? What if the agent's practice run executes flawlessly?

18:34It doesn't crash Python, it doesn't delete the database, but in the process of doing its job perfectly, it accidentally breaks federal law. or it emails my private social security number to a marketing firm. Yeah, that would be bad. This highlights the critical distinction between state safety and data safety. Precisely. Branching protects the physical state of the machine from crashing, but we must also govern how information is allowed to flow through that machine. This requires data flow control or DFC. Like the tax preparation agent example. Yeah, that's a great one. Say I employ an agent to calculate my business deductions by scanning my credit card receipts.

19:10To do this legally and safely, it has to follow three strict rules. First, it must be private. It can only send aggregate summary data to my accountant, never individual receipts. Right. Second, it must be grounded. It cannot hallucinate a fake purchase to boost my deductions. And third, it must be law abiding. Under U.S. tax law, you can only deduct 50 % of the cost of a business meal. Those are complex semantic rules. And the instinct in the tech industry right now is to just use a massive language model to police the agent. Oh, like feeding the agent's final query into a frontier model and asking, does this violate any privacy or tax laws?

19:47Exactly. We ask the AI to police itself. And I'm guessing it fails disastrously. Disastrously. When they tested frontier models on enforcing these policies, it took seconds to evaluate each query. And the models only achieved an F1 accuracy score of 0.4. Break that F1 score down for me because that's a pretty specific statistical term. Is a 0.4 just a 40 % grade? It's actually worse than a standard 40 % grade. An F1 score balances false positives against false negatives. Okay. In standard accuracy, an AI might look great simply by rejecting every single query, safely blocking bad actions, sure, but rendering the system completely useless.

20:26Ah, I see. The F1 score measures whether the AI can accurately catch the violations without blocking legitimate work. A score of 0.4 means the model is essentially guessing. So it is too slow for real-time operations and completely unreliable. Exactly. You cannot base your company's legal compliance on a VIDE check from an LLM. So if the AI can't reliably enforce the guardrails, who does? The environment itself. The DFC policies must be baked into the fundamental infrastructure. They identified two vital properties for this. The first is that policies must be optimizer invariant. Optimizer invariant, meaning what?

21:02This means the rules are mathematically enforced deep inside the database query engine, regardless of how the database chooses to execute the search. And does that slow things down? Not at all. They tested this on massive, highly complex enterprise database searches, and the internal enforcement ran with near zero overhead. Wow. It doesn't slow the system down, but it mathematically guarantees that the query engine will physically refuse to expense 100 % of that business meal. But wait, the data doesn't stay confined to the database query engine. We just talked about the dependency web. Precisely why the second property is required, representation invariance.

21:39The data flow control must track the restrictions seamlessly as the agent moves the data across different formats. How does that actually work mechanically? Like if a piece of data leaves a secure SQL database, gets pulled into a temporary Python data frame, and then gets pushed out to an external tax API, how do the rules follow it? The environment acts like a digital die pack. Oh, interesting. Yeah. When the data is tagged with a restriction-like law-abiding 50 % limit, that tag is attached at the memory level. At the memory level. Right. When the agent pulls that data from SQL into Python, the underlying operating system tracks the memory pointers.

22:16The restriction tag travels alongside the data into the Python data frame and from the data frame to the external API call. So the environment enforces the policy at every boundary constraint. Exactly. Ensuring the agent physically cannot leak restricted data or break compliance, regardless of which tool it uses. So what does this all mean? If we step back and look at the whole picture we've mapped out today, we started with a genuinely terrifying vision. We really did. We picture autonomous AI agents let loose in our systems, given the power to write data and make changes, capable of causing company-ending damage due to a single hallucination.

22:53But the solution is not to retreat and keep AI locked in a useless read-only box. Right. The solution is transitioning to a virtuous flywheel of eugenic data environments. We amplify the AI's capability by actively retrieving, managing, and eliciting latent data, literally teaching it the unwritten rules of our business so it doesn't fail out of ignorance. And simultaneously, we bound its risk. Exactly. We implement 66 millisecond branching so it can practice in harmless alternate realities, and we embed strict data flow controls at the memory level so it is physically incapable of breaking compliance.

23:30This raises an important question, however. What's that? The ultimate endgame here is not just about building smarter models. It's about building digital ecosystems that improve themselves through use. Like a compounding effect. Yes. Every time an agent elicits a piece of unwritten latent data, or every time it optimizes a new query path safely inside a sandbox, the environment itself becomes more robust. It becomes a smarter infrastructure for the human workers and for the next generation of agents. It's a total paradigm shift. We aren't just making the workers smarter, we're fundamentally upgrading the intelligence of the office itself.

24:04Which leads me to a final thought for you to mull over. If we successfully build these agentic data environments, environments where AIs can branch reality, simulate thousands of complex outcomes in milliseconds, and perfectly obey all these embedded physical laws of data flow, What happens when those digital training sandboxes become so incredibly complex, flawless, and self-optimizing that the agents operating within them can no longer tell the difference between the simulation and the real world? Now that is something to think about. Something to keep you up at night while your autonomous agent finishes your tax returns.