Karpathy’s autoresearch could make scientists of us all

0:00Today, I wanted to talk about a little tool that I came across about a month ago that has really changed the way that I do some of my work. And that was a tool released by Andrej Karpathy. Now, what did Karpathy release? He released this thing called AutoResearch. It's a piece of open source software. It's only 600 lines of Python code. It's already had 57 ,000 stars on GitHub. And what auto research does is it enables AI systems and you running an AI system to automatically conduct research. I said, look, this could probably work outside of the machine learning domain. Could we find some other way of defining the objective, right?

0:45Why does it have to be the loss function in an ML product? Couldn't we define a business objective? So my new setup is this new version of auto-research. And what that allows me to do is run this iterative loop, this scientific loop, hypothesis-driven testing automatically on all sorts of commercial, intellectual, academic, theoretical problems that are relevant for us. This is a reduction in the cost of the scientific method. You know, I'm applying this scientific method now to questions that benefit from it where it would have been too expensive previously. And science is the best method we have found for producing knowledge.

1:26And we have given that method now to LLMs at very, very low cost. And I get to choose what they investigate. Well, I mean, for now.

1:43I want you to zoom back and understand what this is about. This is about addressing the best strategy that we as humanity have for producing knowledge. That strategy is called science. And science is the act of coming up with some kind of hypothesis and iterating through a beyond that hypothesis. And it's science that breaks us out of the Malthusian trap. You know, prior to having regularized science, we effectively hemmed and hoard with natural resources and population. And science allows us to get more from less. The scientific method is the thing that allows us to do that really, really reliably.

2:30So that's the big picture. Now, Andre Carpathie is one of the great AI researchers. He's much more than that. He's also an educator in AI. He is also somebody who builds practical tools. And a few weeks ago, he launched something called auto-research. The thing that auto-research does is it allows you as a human to conduct certain experiments. And what you do as a human is set a strategic direction. You set some constraints, you define what good looks like, and then you give that as a set of guardrails to your agent. And the agent does lots of experimentation and has full autonomy as long as it stays within its lane.

3:15so there's this autonomous improvement loop within auto research you change one thing the agent picks a hypothesis it sees how well that hypothesis performs it measures that and if it's better it keeps that as a new baseline and it moves on to another hypothesis and if it's not better it discards it you've solved there the principal agent problem right the human owns the objective, the function, and the strategy, and the agent owns the execution. And the human can't get annoying and interrupt the agent, and the agent can't ever get too big for its boots. Now, the way that Karpathy designed this was that every experiment takes about five minutes.

4:00So you have up to 12 experiments an hour, which is pretty fast. And you can run hundreds and hundreds of experiments in just a couple of days. Now, that happens to be a lot. I don't know what your baseline is, but it turns out that 700 ml experiments in two days is a lot. And what he discovered was 20 genuine improvements and an 11 % speed up in the one thing that he was trying to do. Toby Lutke, who is the CEO and the founder of Shopify, which is that big e-commerce platform, adapted this and he was able to develop as a consequence a small machine learning model that beat ones that were twice the size.

4:38What we've done within auto research is solve these two really interesting problems. The first is this small automation in the production of knowledge. And the second is the agent control and harness problem because the way you design an auto research is you give a specific objective the agent has to optimize for and it can't argue with you. It's a little bit like the human effectively determining where a car wants to go and the AI being the engine within the car that just churns away. Now, this is all very technical. How many of us are really trying to optimize machine learning? Probably not many.

5:20But I looked at this and I took this product and I said, look, could we find some other way of defining the objective? Couldn't we define a business objective in some kind of way that was measurable and tangible? So if the objective was to come up with the headline for one of our essays that was more likely to get clicked, could we have three or four judges, synthetic judges, that would judge that on three specific criteria and simply take the overall sum of their nine judgments, three judges, three criteria, three times three is nine, and try to maximize that over the course of an iteration process.

6:03So I constructed this version of auto-research. It is not as technically robust as what Carpathie had, but it had the same design principles. I, as a human, set the constraint and the experimentation takes place within the loop. And what I found was a really, really impressive piece of software. I found I was able to apply this to headlines for articles we were writing, but more importantly, I was able to find it for a thesis. And I was able to say, look, this is a thesis I'm exploring. Let's have the Oracle made up of synthetic judges and give some judging criteria to the Oracle. The way that I have built my version of auto research, this is version one, is that you can either specify what you want the criteria to be, or you can leave the AI system to come up with reasonable criteria.

7:01And then it will run through loop after loop after loop. And I will get an email that will show every iteration of the sharpening of a thesis and why certain decisions were made. And in one case, for an element in my new book where I wanted to kind of get a really strong argument in a particular area, we ran through about 19 iterations. And in fact, it was iteration 17. That was the one that I thought did the best and scored the best. Now, what are the oracles doing is the oracles are actually scoring all of this out of 10. And you can see the quantitative leaps that are being made. We've gone from 4.6 to 5.2, from 5.2 to 5.9.

7:41We went from 5.9 to 5.7. So we will go back a step and we will do this process again. I have found that this version of auto research works across a whole range of business problems, provided that you're able to crunch down that problem into a single scoring metric, a single scalar. Now you can hear that and you can say, well, listen, there are all sorts of problems with that approach, right? The first approach is that can you really take a complex business problem and simply give it a single number? I mean, there are so many factors involved. And the answer is, you are, of course, simplifying.

8:23As any reasoning or decision tool involves, it involves some degree of simplification. I mean, that's the nature of it, because the decision you have to take is A or B, but the world is very multi-hued. There's a second problem. And that second problem is one that if you've ever worked in machine learning, or in life, in fact, you'll have come across, which is that what's expedient is not necessarily what is best. So in machine learning or mathematical terms, that's about finding a local minima or a local maxima, rather than the global minima. What do I mean by that? What I mean is that if you think about a problem and you think about what the very very best possible practical solution might be you might never see that solution because you find a good enough solution for it if you imagine a a landscape which has got some valleys in it and what you're trying to do is find the deepest possible valley you might be in quite a deep valley is that the lowest point of all the valleys in this landscape or have you just got to one point that's called the local minima problem.

9:34And of course, an auto research and auto loop set up like this can run into that local minima problem. I found that I was running into that local minima problem, or rather, I was skeptical of the quality of the results that I was getting back. And so I pushed quite hard to figure out how can I get out of those local minima? How can I give myself more confidence? So I took my version of auto-research. Now it has exactly the same process archetype as Karpathy's. The human sets the objective, the human sets the guardrails. Slight caveat, I actually allow a separate AI process to set the objectives if I'm feeling a bit lazy, which as those of you who know me will know is a lot.

10:18And then the AI executes within those guardrails and it executes a loop. It executes little experiments. So if it is executing something around a thesis, it might say, let's change a particular word from could to will and does that strengthen the thesis or not? And then the Oracle evaluators give that a score and we decide whether it's improved or not improved. So from that process, I wanted to figure out how to avoid the local minima. If you're doing a commercial exploration, like what should the pricing of this partnership be? Or what's the value proposition we're going to take to this client?

10:56You can often get something that is bland and safe and over-optimized. And so what I built was an escape harness. The escape harness is an additional layer that starts to throw in some random behavior, just tries to push the system to a different part of the landscape. Because if we were already at the global minima in our explorations, we would eventually find our way there once again. So you sort of just kind of throw the problem further out. If it converges, we were at the global minima. If it doesn't, we found a better place. It's a little bit like evolution's sudden mutations. And that works really well.

11:40It's worked well in arguments in speeches that I'm giving. If you read our piece on helium shortages at the start of this week, one part of the contribution to the reasoning was the work that I had done with my auto research on the question of helium shortages. so my new setup is this new version of auto research i'm going to tell you what it's called in a second and what that allows me to do is run this iterative loop this scientific loop hypothesis driven testing automatically on non-ml problems but on all sorts of commercial intellectual academic theoretical problems that are relevant for us what we do with this escape harness is we avoid the problem of the local minima.

12:32A quick note, if you want to support us in bringing more of these conversations to the world, please consider subscribing to the show.

12:43Now, I've called this piece of software Auto Wolf, and I just want to explain why it's called Auto Wolf. In the 80s, I used to watch lots of terrible TV, and there was a really, really terrible show called Auto Man, which was about a man who somehow had superpowers and would turn into a car, which was slightly absurd, but it stuck in my mind. It ran at the same time as another terrible TV series called Airwolf, which was about a very powerful helicopter. So I took auto from Auto Man and wolf from Airwolf and put them together and called it Auto Wolf. And Auto Wolf now is a bit of code that my open claw agent, Armini Arnold, or one of the open claw agents that work for it can call on if we need to do some reasoning in a particular area.

13:30And it has all of those looping capabilities, but also that escape harness in place to prevent the local minima problem. There are some practical learnings from all of this. One is that the loop can find moves that you wouldn't make. Some are better, some are worse, and you have to judge. The cost is trivial. It's dollars, maybe tens of dollars, often on work that you've procrastinated on for a long time. You do need some kind of stopping criteria. So I tend to have this thing stop at 20 iterations. You do want to have the discipline to check where the thing is going after five or six iterations.

14:16So normally the way I have it set up is every five iterations, there's like a bit of intervention that I would go in and make and say this is going in the right direction, or this is so bland, I just need to stop it. The pattern works, but it doesn't work on everything. And it's part of a much larger architecture. So within Exponential View, we have a sort of a ladder of reasoning architecture tools that we use as part of our overall reasoning architecture, which starts with single-shot expert panels where we can draw personas in from our persona library of hundreds and thousands of personas. We then have kind of the traditional one-shot auto-research, which is built within large language models top to toe, which is iterative and pretty good.

15:06There's also now AutoWolf, which has a bit of code to make it a bit more reliable, but also has the escape harness. And then beyond auto-wolf, there are more sophisticated, more expensive methods that we will use in order to, you know, make sure we're thinking things through. There are still kind of interesting challenges that come out of all of this. If the outcome is unmeasurable, you can't really use a process like auto-wolf at all. I mean, And auto-wolf will need that objective measure that gets syncretized by the oracle and the judges that you've created within the oracle. If those metrics are contested, you run into the same problem.

15:48If there's complexity and path dependence, again, you've got something that is too complex a problem to put into this. There's something else that emerges from all of that. And it's true within our overall reasoning architecture from the expert panels to auto-research to auto-wolf and to the things that sit beyond, which is that the human interaction is still absolutely critical. I mean, look, is it garbage in, garbage out? Well, to some extent it is, right? If I kind of bark something that's incoherent, we're going to optimize on something that's incoherent. but you have to judge and you know your role moves from doing the work to judging the work we're talking a lot about this in terms of how do humans fit into a world with this kind of machine cognition machine intelligence and people talk about it being to do with validation or we talk about it to do with verification this is judging i mean verification is one subset of judging.

16:48And judging and validation are similar, but they're not identical. And by judging, what I mean is that I have to look into these thinking traces that are quite long documents. I mean, every round is maybe a thousand word markdown document that comes out and have a look at it or a scan and say, that's heading in the right direction or that's not heading in the right direction. Now, in truth, I'm used to doing that anyway, right? These little gray hairs on my beard are a sign not of too much sun exposure, but of the fact that I've been working for 30 years, which means that I'm used to getting people who are working for me to give me interim updates and for me to course correct them.

17:26Sometimes it's a nudge here. Sometimes it's a nudge there. Sometimes it is the escape harness. We've started down this path. Actually, it's clearly not the right one at all. Let's go somewhere else. So I'm used to that method. It's just that I'm having to do it at a much higher cadence than I ever have before. So if I run an auto-research prompt on something that matters, I will often find myself in the morning, the first 30 minutes of the day, going through things that would run overnight or that have backlogged from the day before to say, are these right and do we need to move them on in any way?

17:59And then you get to the next part of the bottleneck, which is the pace with which we're making decisions is much faster than the rate at which we are used to making decisions and therefore acting on those decisions that come through. And we are learning how to make use of all of that and to make sense of it. But that's what this iterative loop has enabled. Because when you run a Carpathia auto-research in this loopy economy, the loop economy, you get through something that was going to take a week in perhaps an hour. And I've certainly found that similar processes happening in the way, the types of problems that we would look at, what that means is that there's a whole class of decision making, which might have been made, to put it euphemistically, heuristically, like, oh, I just kind of made it up.

18:50Now go through some kind of soak testing in a way that is robust, and in a way where I have been forced to explicitly state what the objective is. That in of itself is a very, very useful piece of self-reflection to help me think about the problem to hand. I think this is a really interesting product capability that Carpathie released and certainly it's become part of my toolkit in the last three or four weeks. because I do turn to it and think, okay, let's go off and push that through auto-research. Not for everything, not everything benefits from something like that. What I love, though, is that this is a reduction in the cost of the scientific method.

19:37You know, I'm applying this scientific method now to questions that benefit from it where it would have been too expensive previously. And science is the best method we have found for producing knowledge. and we have given that method now to LLMs at very, very low cost. And I get to choose what they investigate. Well, I mean, for now, those are human decisions that I make. And of course, those of you who know me, who listen in on this know that I'm always experimenting to see to what extent the agents can help me set up the right experiment to give them the right objective the next time I run that loop.

20:15As I experiment more with Armini Arnold and the agents that work for it, I will of course come back and report my findings to all of you. If you're interested in me putting in the effort to make AutoWolf available on GitHub, then on the Sunday newsletter when it comes out, please comment again, leave the specific thing that you're interested in looking at and that will give me motivation to find the time to do all of that.

20:51Thanks for listening all the way to the end. If you want to know when the next conversation is released, just hit subscribe wherever you're listening. That's all for now and I'll catch you next time.