AI Agents: Substance or Snake Oil with Arvind Narayanan - #704

0:00And here's the fundamental paradox with agents. The capability of agents, I think, is already in some ways mind-blowing. You know, if agents could do reliably in the real world, in the hands of consumers, everything that they're capable of, it would truly be an economic transformation. But even if they're going to fail 10 % of the time, it's a useless product because no one wants to have an agent that orders DoorDash to the wrong address 10 % of the time, right? These are the kinds of failures that consumers are actually reporting.

0:44All right, everyone, welcome to another episode of the TwiML AI podcast. I'm your host, Sam Charrington. Today, I'm joined by Arvin Narayan. Arvin is a professor at Princeton University. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Arvind, welcome to the podcast. Hi, Sam. Thanks for having me on. I'm super excited to jump into our conversation. We'll be talking about some of your recent work exploring AI agents and how we should be benchmarking and assessing their performance, as well as your recent book, AI Snake Oil, which I should mention you co-wrote with Sayash Kapoor, who we spoke to on the podcast earlier this year, talking about the risks of open AI models.

1:35And I think you were a co-author on that work as well. Yeah, we've done a bunch of research together looking at the risks of AI in various ways. Yes. So really looking forward to jumping in. And let's get started by having you share a little bit about your background. Sure. I'm a computer scientist. I'm also the director of a center here at Princeton called the Center for Information Technology Policy. So I split my time into a few buckets. One is technical AI research. These days, it's been primarily in the AI agents topic. In the past, a lot of it has been on AI bias. And then a second bucket is advising policymakers, you know, where AI is going, what sorts of guardrails are needed, policy on open versus closed foundation models, those sorts of things.

2:23And perhaps the third bucket is breaking this all down for a broader audience beyond AI researchers and policymakers. And that's kind of where the book comes in. And we have a newsletter as well, AI Snake Oil. So I spent quite a bit of time doing that as well. Yeah, talk a little bit more deeply about the way you kind of craft your research agenda. What are some of the problems that you've been digging into? What are some of the ways you formulated them? And how do you kind of pull all that together into a broad research agenda? So when we look at AI, one of the things that I think as academics we can contribute is doing things that companies are maybe not going to do, even if they're capable of doing, right?

3:15Because we don't want to be competing with Google or Microsoft or OpenAI. And so I think benchmarking is one area where academic research can really contribute. and more rigorous evals are needed now more than ever. So as we moved from traditional machine learning models to foundation models, evaluation became a lot harder, right? Because with a traditional machine learning model, it's built to perform one task and that task is often something really simple like handwriting recognition. So, you know, MNIST is a pretty decent benchmark for handwriting recognition. There are limitations, you know, it doesn't measure distribution shifts, et cetera.

3:55But that said, you know, it worked well for decades. But then you get to foundation models. These are models that are supposed to be able to do everything to some degree. And so benchmarking started to become really messy. I mean, we've all heard that story many times. You know, every time a company releases a model, they pick some set of benchmarks and their model looks good. But it turns out they prompted it in different ways or whatever, right? So, and then the vibes don't really match the benchmarks. So that was already a problem. And we're expecting it to get harder as the models are getting trained on training data that they've generated.

4:34Exactly, exactly. And another thing that makes it harder is when you get from models to AI agents, because agents are supposed to be able to do things in the real world. And, you know, if you have an agent that is supposed to book flight tickets, you can't actually ask it to book flight tickets, right? And so you have to create some simulated version of that. And now you have to worry about, does your simulated version actually model reality? Or are you encouraging developers to build brittle agents that will do well on your benchmark, but completely fail when you put it on a real website? So those are some of the challenges that have motivated our work.

5:10And this kind of ties in neatly with our other gig that I mentioned of calling bullshit on some of the overhyped claims. Right. So that's what AI snake oil is about. So can we can we actually take that premise and try to make it rigorous, build better evals and actually use that to push the community forward towards agents that work better in the real world and not just do well on benchmarks? Yeah, yeah. And it's interesting that kind of agents is one of the areas that, I don't know, I guess it strikes me as like this nexus for both promise and what many are hoping to enable with AI, even historically.

5:55And bullshit, because there's a lot of inflated expectations, I guess we can call it that. Um, what's your sense of, you know, where we are today with regard to kind of the capability of agents? And I guess I'm mostly interested in like, do you, when you look at the technology, do you see like fundamental or where do you see fundamental limitations such that, you know, we'll never catch up to the hype, you know, versus, you know, it's just being early and the tooling needing to evolve and the benchmarks needing to evolve and all that? How do you think about that landscape? That's a great question.

6:37And here's the fundamental paradox with agents. The capability of agents, I think, is already in some ways mind-blowing. And I'll put it the following way. If agents could do reliably in the real world and the hands of consumers, everything that they're capable of, it would truly be an economic transformation. And this is what I think a lot of developers have not fully recognized, right? This is what we call the capability-reliability gap. And when you look at a lot of the products that have flopped on the market recently, like the Rabbit and so forth, I mean, you know, I get the concept, right?

7:13And sometimes these magical experiences do work, but even if they're going to fail 10 % of the time, it's a useless product because no one wants to have an agent that orders DoorDash to the wrong address 10 % of the time, right? These are the kinds of failures that consumers are actually reporting. So that's one thing. The capabilities are really strong, but how do you get that last mile? And that's the same issue we had with self-driving cars, for instance, right? We had really good prototypes, you know, 20 years ago, and that's why developers and CEOs perpetually thought that self-driving was two years away.

7:49I mean, we're finally there now, right? Waymo is offering rights to the public in several cities, but it's taken decades to get there. And maybe, you know, maybe agents are going to take that long, but maybe we can accelerate that through better rigor in our research and development. There are, I think, a bunch of other challenges with agents, but reliability is the number one challenge. Mm-hmm. Yeah, when I hear that, I think a little bit about an argument that I see raging on LinkedIn and X about code generation. And, you know, it almost strikes me as like two sides talking past one another in the sense of, you know, anyone who's used it, particularly on something that they're not intimately familiar with, like is astounded because it's like helping you write some app in some language that you've never programmed before.

8:39And that's amazing. Um, but I think the calls for, you know, broad studies that look at the impact over many developers and, um, you know, the degree to which the bugs that it introduced, uh, you know, if you're not trying to build a production system, those bugs don't matter. But then when you're really talking about productivity on the economic scale, they do, uh, those are valid as well. Right. That's exactly right. That's exactly right. Yeah. And I think you really put the finger on it. There's a difference between coding and software engineering. And I think that's a big part of where the gap is.

9:14So interestingly for me, as a computer scientist, a researcher, I write a lot of code and most of it is not production software, right? So actually for me, agents for coding are extremely valuable because a lot of my code is something that only needs to run once because I'm just testing out some interesting idea that I had. Is this a promising research direction, right? And also even in my personal life, let me give you this example. This is one of my beautiful moments that I had with my kid, which was a positive use case for AI. She's five years old. I was teaching her to tell time. And you might be wondering, what does this have to do with agents and coding?

9:50But it does. And that's the crazy thing. Engineers always make things more complex than they need to be. Well, but yeah, but this wasn't complex. And it didn't require any engineering skills. So I drew a bunch of clocks for her to teach her how to do it. and she was really enjoying it, but it kind of got tiring to keep drawing these clocks. And, you know, I pulled up Claude and said, make a random clock generator app. Right. And so this is something that I call a one-time use app. I don't know if there's a standard term for it, but I find myself doing this more and more. And it produced the app within a minute.

10:23I gave it some modifications that did that. And then I played with her for five or 10 minutes using this app, having it generate random clock faces and asking her to tell the time. And she really got it. That's awesome. Yeah. Within those 10 minutes, and it would have been much more annoying for me to have to draw 50 different clocks on a piece of paper. And then I've never used the app again. So before AI, it would have been crazy to try to sit down and write an app for this. It would have taken two hours. But now I can do this with AI, and it's not a production software engineering scenario, so it doesn't matter if there are bugs, etc.

10:58But you mentioned self-driving cars as an analogy. And early on, one of the conversation points was always that the corner cases are too vast and we'll need to constrain the infrastructure, like build lanes for self-driving cars or that kind of thing, have them only operate as airport buses or what have you. I think we're a little bit beyond that to some degree, but I'm wondering if there's an analogy for agents that, you know, is the idea of a way of like, I guess, guardrails or like constrained environments for agents. I think in the conversation with Sayash, we talked a little bit about the idea of just how much risk is involved with allowing them to actually interact in the world.

12:01And I'm wondering how you see that playing out safely. Yeah, for sure. So there is a strong analogy here, and this is actually a big part of our next research project, building on the AI agents that matter paper. So we're making a bet on what people are calling verifiers. So the idea here is that if you have a coding agent, then a set of unit tests is a verifier. So you have the agent write code, see if it passes the unit tests. If it doesn't, you can throw it away and start again or give it feedback and ask it to debug it, et cetera. So a verifier is a kind of guardrail. And this relates to OpenAI's Strawberry model as well and other things that other companies are working on.

12:46from what I understand, the way that they have trained Strawberry is, or, you know, a one preview, yeah, whatever they're calling it, is having these kinds of domain-specific verifiers and then, you know, using reinforcement learning, if it didn't ultimately lead to a passing score from the verifier, then that's, you know, negative reinforcement, that sort of thing, that's great. So this leads to a general model that can work well in a bunch of reasoning domains, But still, it doesn't give you that 99.9 % reliability. So what we're asking is instead of a general model, if you want to make it domain specific, if you want something only for coding or if you want something only for web navigation, right?

13:28So can you build really strong verifiers that you can be confident with 99.9 % accuracy? It's going to be able to tell you if the code is correct or if it navigated the website correctly. And can you then use that to build domain-specific agents that can actually work reliably? It sounds like verifiers in this context is a design time constraint as opposed to envisioning some kind of runtime constraint. I would always say for a long time that agents is a political problem as opposed to a technical problem in the sense of for many years and to this day, companies actively try to restrict crawlers from accessing their websites.

14:15They are selective about what they make available via APIs and to who and their licensing agreements. and part of the promise is that I have this agent and it'll be able to do anything that I can do, but there are legal and other reasons why we haven't been that. In other words, part of what needs to happen is kind of changing the way we think about machines accessing systems. And I wonder if your research has kind of come across that idea. Do you think that the kind of momentum of AI will make some of those changes societally or will we still have these kinds of issues? There's a case that the momentum will overcome some of these barriers.

15:07There's also a case to be made that it will make things worse because everybody who's not a big AI company is just so terrified of AI companies capturing all the value basically. And every website or app now just becoming a sort of backend data provider or service provider. And I wonder if this is one of the reasons why OpenAI's GPT's feature didn't really take off where they wanted Expedia or whoever else to be essentially hooking into ChatGPT so that ChatGPT would be a universal interface to access all of these. Right. So that that, you know, that's that was an attempt to realize the vision you were talking about instead of having to build an agent that can navigate the web generally, you know, bring the web to the agent.

15:57Right. And I don't know why it hasn't worked. I think maybe it's on the user side. Maybe the user experience wasn't great or maybe it's because they didn't get as many providers to play nice with them as they wanted. And we're seeing that a little bit when it comes to web crawling, which, as you mentioned, there was a study recently showing that over the past two years, the percentage of websites using robots .txt to stop crawlers from accessing them has shot up dramatically. Interesting. I hadn't heard that. Yeah. Yeah, I think Shane Longpreay was the first author. Yeah, so I totally agree. It's as much economic and political as it is technical.

16:41I think for agents to work, we're going to have to figure out who's going to capture the economic value that they create and how to do that in an equitable way. You know, we jump right into this conversation as I often do, but defining agent is still an open question in many ways. When you think about agents or agentic behaviors, like what are all the elements that come to mind for you? So before I talk about those elements, let me say for a second, you know, one might ask genuinely, you know, is this even a meaningful term or is it just marketing hype, right? And I think that's a fair question to ask.

17:22But here's why, I mean, there is a lot of hype around agents, but here's why I do also think it's a meaningful term. Unless one thinks that companies are going to be able to scale models all the way up to AG high, and I don't think that's going to happen. Right. And so if that's not going to happen, then it must inevitably be the case that to take these models and to do more useful things with them than they can do just through zero shot prompting, you have to build more complex scaffolding. And, you know, you can call it whatever you want. You know, it needs some term. And I think for better or worse for these compound AI systems, as some researchers call them, which is also a fine term.

18:01but the more well-recognized term is agents. So I feel it is something meaningful. So in terms of how to define them, we looked at a bunch of attempts to define agents. And it seems like people have a certain set of factors in mind. One, how complex is the environment in which the system is asked to operate? So a chatbot operates in a really simple environment compared to a web agent or some other kind of agent. and how difficult are the tasks that it's asked to do? Are there multiples, take holders, whether people or other agents, is it asked to operate over a long timeframe? So that's one set of factors.

18:43A second set of factors is, is the user basically babysitting it or is the system given a certain amount of autonomy based on high-level goals to go off and do things in the way that it thinks is the best way to accomplish that goal. So that's the second factor. And the third factor is the design patterns themselves. Is this, you know, does it go just beyond an AI model? Does it do tool use, for instance, or reflection or planning or those types of patterns? So I think there's no strict binary dividing line, agent or not agent. And I think the more factors that a system has, the more agentic it is.

19:25Yeah, I think one of the interesting things and perhaps the central thing in the Agents That Matter paper is this idea that we can look at agent performance in different areas of capability and kind of law these agents. But unless we look at the cost and the efficiency of their operations, it's kind of meaningless. Is that what you're trying to communicate with that work? That's exactly right. So many years ago, Google's alpha code paper showed that if you can just repeatedly sample solutions from a code generating agent, and then you had some Oracle, maybe unit tests or whatever. and that let you, as long as one of the generated solutions was correct, you could somehow pick that solution, then you could just keep doing this and increasing accuracy.

20:21They stopped after one million attempts, but even up until one million attempts, the accuracy kept going up, right? Starting from a really weak model by today's standards, they could just keep increasing the accuracy. So if we don't measure cost, what does it mean to say that you have state-of-the-art performance, right? So you could always just invoke the model more and more times. It's like teaching a kid that there is no biggest number because it's always that number plus one. So it's just a basic insight, but somehow I think people were not willing to admit this because it's just very nice to have this abstraction of a one-dimensional leaderboard.

20:59And the minute you say, no, it's at least two dimensions, you have to consider cost. It's not a leaderboard anymore. You have to talk about what's the best model for this cost budget or whatever. And it's messy to talk about dollar costs, right, when you're trying to figure out what's the best model. So what we say is, no, we can, you know, let's plot these agents or models or whatever on a Pareto curve. One axis is the accuracy and the other axis is cost. And instead of a one-dimensional leaderboard, maybe these Pareto curves are what we should be measuring when it comes to benchmarks. and in the last three months, this has become much more common.

21:35It's almost become the default way to talk about performance now. And then when someone wants to look at this to see if they want to actually use an agent in a particular application, they can think about what their own cost budget is and based on that, how to go about things. And the other thing we say is if what you're trying to do is get the best performance for a particular cost budget, But often the simplest way to do that is just to retry until you get it right, instead of more complex methods like trying to debug the code and so forth. So don't forget brute force is one of the lessons of the paper.

22:10Yeah, I've had that experience personally working with some of the smaller models like Gemini Flash, for example. And I don't recall if currently they have, I think they actually, they do now have a structured output ability. But prior to that, it was like, you know, ask it to give you some JSON and try to, you know, confirm that you received JSON. And if not, just keep trying and eventually it'll give you JSON. And by the way, can I can I just point out my gratification in hearing someone else call it JSON instead of JSON?

22:52I have no idea how consistent I am in the way I pronounce that. And so as a follow on to this idea of kind of cost versus performance, one of the things you're working on now is some benchmarking around agents. Can you talk a little bit about those efforts? So we're taking it in a couple of different directions. One is related to, can we build agents for some very specific mundane tasks that people need to do as opposed to, you know, reasoning is very abstract and very tempting, very sexy. A lot of people want to work on that. But if you've solved reasoning, it's still a little bit unclear what you can actually do with that.

23:42On the other hand, let's take a real problem that many people are struggling with. And as researchers, our mind immediately went to scientific reproducibility. So this is just the idea that when someone puts out a paper and code, other researchers should be able to download that code, set up all the packages, all the messy stuff that's necessary, run that code, and verify that it produces the same results as are reported in the paper. So that's a basic prerequisite for being able to build on that paper, right? And unfortunately, it turns out that so often this basic notion of reproducibility doesn't work out.

24:20It's not because anyone's cheating. It's very rarely that, but it's just that, you know, sometimes it just takes too long and people give up. It's just hard to install a bunch of packages. Other times you have a slight difference in your library version. And because of that, there are minor differences in the code that actually translate to substantial differences in the output, things like this. So it's a very subtle problem. And worldwide, tens of millions of researcher hours are wasted on just reproducing others' code. So our question was, can we automate this? And so my team got together. Zach Siegel is the first author.

24:59And we've put out this benchmark called CoreBench. It stands for Computational Reproducibility Agent Benchmark. And so we sourced this from actual scientific papers, 90 different papers. And this is not just about machine learning. Our papers are across computer science, social science, and medicine. And what we say is, okay, so here's the code, and we give it to the agent at various levels of chaos, if you will, everything perfectly documented as it should be, or we remove the Docker file that's supposed to be there to make it easy to run the code or various things like that. So we simulate the way in which researchers very often don't do all the things they need to do to make it easy for others to reproduce the code.

25:47Meaning so for these 90 papers, you've gotten to the point where they should be machine reproducible and then insert kind of artificial noise to simulate deficiencies. Exactly. Exactly. I mean, they should still be reproducible. You know, a person can do it because they can figure out what's missing and how to fill that in. The code is there, write the Docker files, the requirement files missing, you know, figure that out. Yeah. And this turns out to be surprisingly hard for agents because I suspect, you know, it's hard to pinpoint an exact cause. I think it's a bunch of things, but it requires a lot of back and forth interaction with the shell.

26:26So that requires very long context. Agents actually get confused a lot. They forget exactly what the state of the shell is at any given time. And so that's one difficulty. And then after you run the code, especially in fields outside computer science, it turns out the result is not necessarily in a nicely formatted code output. It's in a PDF. And so now you have to visually inspect the PDF using a vision language model in order to extract what it has actually output. And these are all, you know, hard for a model. They're easy for humans, although mundane, time-consuming, and so forth, right? So that's the kind of task this is.

27:05And this is very different from some of the other tasks, which are more aspirational things like reasoning. And so two main differences, right? One is it's kind of mundane as opposed to some North Star challenge. And the other is if you solve this, you're probably immediately going to save people tens of millions of hours per year. So we think this is a nice challenge. Current agents don't do that well at it, not well enough to be useful. But we can see that even with some simple modifications to off-the-shelf agents like AutoGPT, we can push up the accuracy a lot. So we do think that with a good amount of effort on this, we can get it to a point where we can largely automate the problem of computational reproducibility.

27:50And to be clear, it's not a problem you solve. It's an open benchmark that you're offering to the community to kind of go after this problem. Exactly. I guess I'm envisioning like one possible use is, you know, when you check in your work at a conference, this thing does an assessment for you and gives you like red, yellow, green light kind of thing. But then if it can do that, like why not just fix the problems and put everything into a standard format? Like how do you see it being used? Yeah, yeah, absolutely. Yeah, that's that's the first step, right? So to be able to check if things are reproducible and the next immediate step after that exactly is to fix things.