From Math Olympiads to Navier-Stokes: How Fast Is AI Progressing? with Greg Burnham - #778

26:05Greg Burnham:This was sort of a historic moment, the so-called unit distance problem, which was a problem about how many points you can fit in the plane that are just one unit, distance of one apart from each other. It was sort of like very cool that you could frame it. It's such a simple problem, but there hadn't been much progress on this problem despite a lot of attempts since Erdos had posed it decades and decades before. And it was a serious problem. It wasn't one of these, no one had thought about it. It definitely had received a lot of attention and an AI system solved it. And it was really this, this was like a aha moment of, oh my goodness, like AI can solve problems that humans have cared about, just independently.

26:48Greg Burnham:And we've seen a lot more of that since. And the, you know, with a sort of the recent wow moment being the solution of the Millennium Prize problem. This was, you know, the Navier-Stokes problem. This was a collection of these Millennium Prize problems, a collection of seven problems that mathematicians sort of, or at least one math institute set as like good goals for the new century. Like they were, they were, they're much older than the year 2000, but, but they were sort of canonized in the year 2000 as, hey, like these are worthy targets of entire subfields of mathematics and it would be a big deal.

27:24Greg Burnham:And so one of them was solved. So it's been a hell of a, hell of a ride. The historical context is super interesting. And, you know, we're all living through this trajectory, but to step back and see, how much has happened in the past couple of years is pretty nuts. Yeah. You have to, as one of my colleagues said, you have to not get frog-boiled by it. The temperature is rising and it's easy to, I don't know. Anyway. So there is this interesting gap so far. You might hear the phrase jagged frontier, where the AI capabilities are jagged in the sense that they'll be superhuman on some axis and not superhuman, still inferior in capability to human on some other axis.

28:16Greg Burnham:And that remains the case so far in math. In particular, there's a couple of things. In hindsight, the analysis by mathematicians of the solutions AI has found to these big, important, previously unsolved math problems, it's always, in the first place, it's been nothing too super duper surprising. Maybe there was a direction that humans had overlooked or could have invested more in and AI sort of had the persistence, maybe this is a good word, or the bravery or the temerity to like wade into these very in-depth intricate computations and calculations and sure enough came out the other side with a solution and humans look back and are like ah that's not a crazy direction to try at all and maybe if we'd like gone in that direction we would have come up with this this isn't something that like would be surprising what i'm curious about is are we seeing the AI systems come up with a solution to the problem or an algorithm to create the solution to the problem?

29:35And I think I'm trying to like ask a lot of different things here. You know, one is like creativity. One is maybe levels of abstraction or the way it's like, you know, thinking, abstracting, you know, around these problems. Yeah, I'll kind of pause and let you react to that.

29:55Greg Burnham:Yeah, let me throw out a couple things there. Let me know if you want me to expand on any of them. When humans solve these math problems, they write up the solutions in like papers, proofs that, you know, this is true or this is not true. And those proofs are not usually, like not exactly algorithms. They're written in natural language. They have sort of, they leave the reader to fill in some of the gaps. And AI systems are coming up with just the same kind of thing. Like, it's the same products. There's a lot of complaints about their writing style, but like the arguments, the substance of the arguments really are the thing we expect to see from humans, such that if a human came up with this, and yeah, maybe cleaned it up a little, I don't know, like would be lauded for having made a very insightful connection between two different areas or having gotten an intricate argument just right and balanced different considerations to make everything come together to land the proof.

31:00Greg Burnham:A hundred percent, this is something we're past the point where we can say, well, AI systems are sort of not really doing something that humans would be impressed by on human terms. AI systems are doing things. And is there a way to categorize or qualitatively are we finding that, and I think you spoke to this a little bit, like, you know, is there a qualitative difference between, you know, the best human approach and the AI approach in a sense of like, you know, AI brute forced it. You kind of spoke to this versus like pulled in things from some other domain and like created some elegant solution or are we past that point as well?

31:44Greg Burnham:Here I'd give more nuance. There's an emerging sense of, at least for the moment, what an AI-shaped problem looks like. Maybe I could say AI has two big advantages right now. One is it knows everything. Like it knows all the prior work. It's got the literature memorized. I'm speaking loosely, but this is basically valid. The other is it's extremely patient and persistent. So if there is some intricate calculation you have to do, I don't mean like numerically or algorithmically. When I say calculation, I just mean lots of equations and you have to make all the pieces work out just as a very high level conceptual logical kind of computation, but still something very intricate.

32:38Greg Burnham:They just have the patience to go for that. And if you take the Navier Stokes example, I think it's not a coincidence that OpenAI's reported system that solved it included a swarm of thousands of different instances of AI systems trying lots of different things and sharing ideas where clearly it was able to, like, there's an element of brute force there. So there's degrees of brute force. This is where It wasn't like somehow, you know, you asked it to, like, it wasn't solving this in, by any means, in the dumbest, most low-level way. Like, we're past that point as well. They used to solve some problems in, like, kind of surprisingly low-level, like, grind it out kind of ways.

33:25Greg Burnham:Like, that's not really what these look like anymore. but like the high level, a higher level version of that still is maybe what you might say here. So, and I like the frame of precise and intricate computations here. It's important, I'll say one more thing about the Navier-Stokes solution. It's important to recognize that humans sort of put in place a lot of the structure through previous work that AI then went and as far as we can tell, I mean, humans are still figuring out exactly what goes into the AI solution. But as far as we can tell, it builds on a lot of prior human work. The AI didn't like build it up from the ground.

34:03Greg Burnham:That kind of theory building, as mathematicians call it, we still don't see that very much from AI systems. Last point I'll make before pausing on this is just remember its point in time. Like, remember that trajectory. We've seen very rapid. so mathematicians who are saying we have to adapt our profession to the fact that now you can push a button and get what used to have been a career-making result like you know are also I think cautioning and we can expect the capabilities will freeze right now we we see you know that that linear whether it's linear or exponential as a matter of perspective but it's going up and it's probably like we see no reason to expect it to stop there so this is a big thing Epoch is looking for.

34:47Greg Burnham:Can AI move from intricate computations plus lots of background knowledge to more of a developing new ideas, more from whole cloth? And if so, and that's a big thing we're watching for in our own math benchmarking activities. Yeah, it's interesting to think that at least the way I characterize this as brute force versus pulling in things from other domains those are not mutually exclusive. Like for an AI that has access to all of the research from every domain, an approach that could be very fruitful is to brute force the exploration of adjacent domains and pull that into the solution to a given problem.

35:38Greg Burnham:That's it, I think, a lot of what we see. And maybe just to say the other thing that would really supercharge them and bring them closer to full human parity or even dominating human capabilities would be something like coming up with a new idea. And maybe like in calculus, the idea of the derivative or the integral of a function. That was something that maybe mathematicians in the 1500s had been grasping for this idea, and then Newton or Leibniz or whomever comes up with, like, hey, here's like, I can describe a new thing, a new mathematical concept, and that unlocks a lot for us. That lets us solve problems we previously couldn't solve.

36:24Greg Burnham:And so that's the sort of thing we don't yet see AI doing. It's also the sort of thing that's rarer for humans to do. A lot of human math progress has been, well, I sat down and I put in the work and I did the computation and I worked out which was the dead end and which was the good path, and here's the answer. Now, XYZ is true. I've proven it. Or I studied one area and I noticed something and I thought it might be applicable to this nominally, superficially different area. Humans do that sort of stuff all the time. Maybe that's 90%, vaguely speaking, of human mathematical work from what I hear from mathematicians.

36:58Greg Burnham:The new idea stuff is rarer, so maybe we should expect it to be not the first thing AI gets good at. But so far, we don't see AI really doing that yet. How easy or difficult is it to define a new idea in such a way that we can easily distinguish them? Like, you know, if we think about kind of the evolution of this discourse around like, is it AI creative? You know, initially it was like, yeah, it's very creative. It created this like poem and, you know, and pirate like that's creativity. That's, you know, that was a new idea. But that's certainly not what we're talking about when we're talking about like advancing science and advancing mathematics, et cetera.

37:47you know so a how do we like how do we define that but also you know from your perspective as someone who's studying this is it like you know we've got this long line of folks that are coming that says here's this ai with a new idea and you're like yeah well let me evaluate it against my new idea criteria no it's not really that or are we actually like is it pretty clear to everyone that the AI is not having new ideas and like we're waiting for the first person to come and say that AI has the new idea? Like, how do you think about that whole space?

38:22Greg Burnham:Very, very challenging nut to crack. So we, you know, try to approach it from different angles. One thing you can try to say is, I've got a problem. I don't know if a new idea will be required to crack it. And this could be a problem in math or in science or any quantitative area. But humans have tried to crack it. Humans have tried to make this number go up, whether that number is the efficacy of some drug at fighting some disease or something like that, or it's a math result of some conjecture that has stumped mathematicians for a long time. And you can say, okay, well, I don't know if, I'm not sure how I'm going to measure a new idea, but I can at least tell if this problem has been solved or this number has been made to go up or whatever the concrete metric is.

39:17Greg Burnham:And you can then, then you're, you can at least check if that has happened. Now, I was probably a year ago more optimistic that there would be a tighter relationship between math problem that humans have failed to solve and post hoc qualitative assessment of creativity. But, you know, this is sort of the combination, the like, you know, double punch we're trying here. First, you set a target that's hard on human terms that humans care about. Humans have tried to be creative to solve. And then you do post hoc review with experts and say, what do you make of the solution? Why hadn't you found it?

39:58Greg Burnham:No offense. You know, what is in the AI solution that humans had missed and so on. And this is risky, like risky in terms of humans being honest about it, because, you know, your ego might be wrapped up in it, or you might, once you see the solution, it feels more obvious in hindsight. So would you have described it? Like, very risky. But you still, you try your best. And, you know, you might say things like, well, how, like, here's an interesting approach that we haven't operationalized in detail, but it's a nice heuristic, I think. How big a hint would a human have needed to solve this in hindsight?

40:37Greg Burnham:So, for example, that unit distance, a very prominent mathematician, Fields Medalist Timothy Gowers, did this sort of exercise on this unit distance problem once AI had solved it saying, how big a hint and how specific of a hint do I think a mathematician would have needed now in hindsight to solve this problem? He came away with an interesting exploration of, well, actually AI sort of disproved this conjecture as opposed to proving it true. It said, actually, it's false. Even that is a huge hint. Humans had mostly thought it was true. They were wrong, but they'd been trying to prove that it was true instead of looking for a counterexample.

41:17Greg Burnham:So even just the hint of try to find a counterexample is pretty big. And then there was like one more piece. And he was like, I think, though, very hard to say, we can't really do the counterfactual experiment. I think with just that, with like this sort of two part hint, it would have been possible for like a human would have been like, oh, okay, I know what to do now, blah, blah, blah. And like probably would have at least hastened a human solution. So you can do things like that, maybe still very qualitative, but you try to be honest with yourself and be rigorous. And you could imagine doing a more rigorous version of this experiment.

41:48Greg Burnham:And anyway, things like that. But then the fact is math is still, for the most part, a sort of discipline removed from physical real world impact. Not always, but its impact is often hard to predict. There are domains where impact is not at all hard to predict. Like I'm in the chemistry lab and I'm trying to improve the yield of my synthesis process. And if AI makes incremental improvements. I expect incremental yield. And if AI suddenly comes and says, no, no, no, you're doing it all wrong. Let me like redesign your whole thing from scratch. I've got like a, you know, maybe creative idea. Well, at some point you don't care if it's creative or not in the qualitative sense.

42:26Greg Burnham:You just care that you saw discontinuity in the yield of your chemical substance or whatever. And that's, you know, that's a bigger impact for the world anyway. What I took from that is that we filter on the problem and how hard humans have kind of banged their heads against the problem. Then we look at the results that AI has come up with and try to kind of subjectively say, you know, is this based on a new idea or is it more brute force or something else like. But it's kind of imperfect and messy and like we're not really sure, but we're we don't think that we've come up with, you know, great examples of AIs coming up with you know, quote unquote new ideas.

43:17Greg Burnham:That's exactly right. One reference point I'd give is way back in, gosh, well, the teens, I forget which year, the famous Go playing system, the game of Go, AlphaGo. I had that thought earlier as we were discussing this. Like, if I remember correctly,

43:37part of the solution was like, Like, oh, that was an innovative move that no human ever would have done.

43:43Greg Burnham:There was a specific one. Yeah, it was a very specific move in there. Yeah. Yeah. They call it move 37 because it was the 37th move in this game where. Yeah, apparently this was a very surprising move. Like you said, no human would have done it in this particular position in this in this game of Go. And I think it's encouraging for our admittedly quite subjective program here that Go experts looked at it and were like, oh, my goodness, that's new. And then it also became clear that that was pivotal, that that move gave the AI system playing Go a decisive advantage. And so that's sort of what we're looking for in math.

44:27Greg Burnham:And at least maybe we have some hope that humans will be able to spot it when it happens. uh but but yeah that's that's what we haven't seen quite yet and again the trajectory is fast the things are always a little more qualitative than you know that you know than they are uh or continuous than discrete uh but you know so anyway this is this is where we are now which is kind of wild in its own right earlier when we were talking about frontier math you kind of characterize it as you know, one to four, I think, and then advanced and we never circled back to your more recent work? Yeah, we've got a couple, two new sets of frontier math problems.

45:08Greg Burnham:We ditched the tier system, whatever. So now these are, I'll lump them together because it really is the, this is our main tool. So we have, you know, our latest iteration on frontier math consists of a bunch of problems that humans have tried and failed to solve. They're curated to be of a special interest to mathematicians, where, and some of them we've characterized even on a bit of a scale from, like, this would be moderately interesting, or this would be, like, maybe what that means is two teams of mathematicians have tried this problem, it's at least 10 years old, but it's not central to any research program.

45:52Greg Burnham:It's just something that people would be happy to get an answer to. All the way up to like a major breakthrough. Four tiers here. Breakthrough is the highest. None of the breakthroughs have been solved yet, but don't get me wrong. If Navier Stokes had been in this problem set, it would surely be a breakthrough there. Regardless of the fact that humans had like done a lot of work and gotten maybe within spitting distance of the solution before AI finished it off. But yeah, we really view this as just a way of finding AI, a way of finding cases where AI might have had a creative idea, because we can tell easily whether it solved these problems.

46:32Greg Burnham:this is maybe the big piece of work on our side, was to make it easy to verify automatically whether an AI system has solved this problem. It's not obvious when you think about it that no human knows the answer. So how do you tell without a human looking if AI has found the answer? But there's a couple techniques you can use for this, and we employ different techniques for this. It can go into that, but it's something of a technical detail. And so when a new model comes out, Astra came out, we hit go. And it's like, oh, it solved three new problems from our list. Okay, let's go look at the solutions.

47:05Greg Burnham:Let's share them with mathematicians. Let's see what's going on here. And then we analyzed that with the help of mathematicians, analyzed that for like, what kind of qualitatively, what kind of solution was this? What does it mean when the models are so good that we have to shift from, you know, these benchmarks that we understand to like, these collections of problems that we just don't have the answers to? It's definitely a phase transition in the science of AI capabilities measurement, which is what I work on. But it doesn't change the game all that much. It just means we have to look for harder problems.

47:44I think I joke sometimes that even certain forms of optimization problems, like allocating scarce resources efficiently, that's a benchmark of sorts that will survive the singularity.

47:56Greg Burnham:Like even if AI is like off going crazy, running rampant in the galaxy, it'll still have to, you know, it'll still have to decide how to, you know, optimize. So optimization is always going to be a benchmark, even if we're superhuman. And basically, it used to just be we knew the answers ahead of time. Now we don't. So it takes this extra trick of, OK, how do we tell when AI has gotten something right? But this isn't insurmountable, especially like I was saying with numbers in a scientific context. It's intuitive that, OK, the best humans were able to do was X. But we just say to AI, hey, try to do Y greater than X.

48:34Greg Burnham:And if it does, those numbers go on and on. So it's no problem. But yeah, you're right to sort of note the phase shift and to be somewhat surprised by it. And is there a methodology, a concrete methodology for assessing and awarding partial credit? Is that part of the way you think about benchmarks in this post-phase shift? Yeah, it's a good question. It depends on the benchmark. So sometimes it's something like, well, you just want to make this number go up and the higher the better. So partial credit is very continuous and very, very natural. in those cases. For math problems, sometimes it's just like, well, what we care about is, is, you know, if X does Y always hold, like, is this, is this always true?

49:21Greg Burnham:And then it's kind of, sometimes humans will find and publish partial results for the AI systems. We usually just say, you know, all or nothing, but we have a large enough collection of such conjectures that, you know, we hope to have some, you know, smooth gradient of, well, okay, this one got two more out of 100 or something. And so it looks like a pretty smooth signal. That's typically how we approach these things. Yeah, across benchmarks. Yeah, maybe what I was thinking was, once you're in the regime of very challenging, unsolved problems, is it possible that AI systems are doing surprising or novel or creative things, but still not quite getting them to the full, the full solution?

50:08And do we care about that? Do we want to track that? And is that part of the way you've constructed the benchmark?

50:15Greg Burnham:It's a great question. It's not central to how we construct the benchmark. It's certainly worth tracking. The problem is it's just labor intensive. You can have AI review the AI. We do this occasionally. Say, which problems did it make partial progress on? What if we let it think for longer on those problems? And nothing yet. It doesn't seem good at understanding how close it is to a solution. Humans aren't necessarily very good at this either. I did want to mention that while we focused on maybe the area where AI is the very best, like has come the farthest, the fastest math, we do see lots of areas where we also think it's important to measure AI capabilities, where the progress is somewhat fuzzier, less clear, and certainly less superhuman, even in some, well, maybe.

51:08Greg Burnham:be. Still superhuman in subways, but the ways in which it's not yet are maybe more obvious and feel less rarefied. What are some examples? A couple examples. One area that forthcoming work we're excited about is just, it's a question everyone must be wondering, can AI take my job? So our job at EPOC often involves many things and all sorts of research, not just AI benchmarking, but other topics about trends in AI and writing research reports and doing data analysis, making infographics that make a point in an easy to digest way. We've been collecting examples of these from within Epoch, where the judgment at the end of did the AI system manage to do this Epoch task is too messy for traditional benchmarking.

52:04Greg Burnham:So it wouldn't be obvious, like, how to say, did it get the right answer or not? It's not about making a number go up. It's not about satisfying some logical chain of deductions. It's like, here's the infographic to go along with this report. Like, does it meet our style guide? And does it look good? And does it, you know, is it better or worse than the human reference that we, you know, one of my colleagues did? um and here we see like an interesting mix of uh some things i think we we sort of all our experience is like yeah if you ask it for a summary of the literature it's going to do a pretty good job it's like very good at that but but there's other tasks where it's um you know hard like creating compelling visuals that follow our style guide or this is a good one coming up with a new short-form data-driven insight.

52:59Greg Burnham:We call these data insights. You can find them on our website. And these, often we try to boil down a single important observation into a single chart and a single sentence. And this is hard, but we don't see them doing a great job at this. Come up with a new project for EPOC to undertake and do a prototype, very open-ended, not so good at this. Like these more open-ended tasks, we do see, again, my guess is if we will do this, go back in time some and see, have models gotten better at this? My guess is we'll see progress. But the level compared to like humans is still a little, still a little meh.

53:41Greg Burnham:I want to give one other example, which is learning on the fly. This is another case relevant for can AI take my job? Because you weren't great at your job on day one, but as you practiced, you got better, and humans have this sort of, they learn on the fly. We take a, we like to measure this for AI. We have a benchmark that's taking a complicated board game. We did a human study for saying, how many times do humans have to play this board game in a row before they sort of figure out how it works and can get a very high score on it? We do the same with AI. We said, play it once, take notes, same AI instance, like play it again, take notes, play it again, take notes, and see how you can do.

54:26Greg Burnham:AI systems have gotten good at this board game, but not by playing it repeatedly. It's more of an intergenerational thing, like GPT 5.6 scores X out of the box and then stays flat on its playthroughs. GPT 6 scores much higher. It's like much better at it, but it's still out of the box. And then it stays flat in its playthroughs and is still less than the top human. So this kind of on-the-fly learning is something we also don't see AI systems as being good enough at, even to just pick up a board game, let alone a messier job, a real-world job like that. So these are sorts of things where we don't see these capabilities and we think it's important to try to catch when they emerge.

55:05My gut on the ladder, the AI learning on the fly is that it's maybe a harness solvable challenge versus a model solvable challenge. And maybe, you know, no one's focused on that. like with the right set of memory structures and like some kind of sidecar heuristic thing like you know we can probably make a big step improvement in an AI's ability to like you know improve game to game but it's not uh it might not require dramatic model changes Do you think about that? And is, do you have, you know, in this domain or in other domains, like formal or structured ways of thinking about, you know, what, you know, model versus harness, you know, granted, this whole model versus harness thing in and of itself is, you know, practically brand new.

56:07but it's proving to be important in distinguishing where these improvements come from.

56:15Greg Burnham:Yeah, definitely. We've done some experiments on this for the board game case in particular, where we've tried a very simple harness, the first party harnesses like Claude Code or Codex. We've tried multi-agent setups. We give them note-taking tools and make it clear, like, okay, your goal is to learn the game so you can do really well in your final plays of the game. So explore, take notes, figure out. We try to prompt them well. I mean, we want to find good results if we can because that's a big deal for the world. The game is a test bed, but real jobs would also benefit significantly. Risks would also go up if this capability, you know, you test in the lab, is it a great hacker?

57:05Greg Burnham:You find maybe it is, maybe it isn't, but you find that it isn't, and yet it can learn on the fly, then okay, once you deploy it, like, look out. So anyway, we've tried pretty hard and have mostly not found anything that cracks this, that cracks this on the fly learning. There's been some general research about this as well, not just us, about, you know, people taught, like there was, people still use these so-called skills where you like just write some notes to an AI system of, look, when you do work I ask you to do, you're going to need to use this tool or produce this kind of output. And like, here's my notes on how you could do that.

57:41Greg Burnham:Well, so you don't have to like learn it on the, like pick it up fresh every time. So use these skills and these show some improvement, but then there's like a plateau, like pretty quickly, at least in the research I'm familiar with of like, you can get some improvements. My senses, this is very loosely speaking, it's like low-hanging fruit like if you're telling it oh you're gonna have to navigate this like finicky website that my company makes me use and like just you're gonna get stuck and like you have to look in this menu and that's where the button you're looking for like straightforward advice that's like obviously useful but when it comes to higher level strategic uh thinking kind of hard to bake that into the harness uh where or or like where where the the the board game plays like like has this, you know, sort of brings this out where it's like, there's not any perfect advice that, like, you have to find the advice for yourself, in a sense, like, you have to learn the game, like, so I will say, if we give it extremely detailed strategy guides for, here's how you beat this game, like, here's everything you could possibly want, like, it's a cheat by humans, like, no one would play the game that way.

58:50Greg Burnham:They do very, they do very well, they do much better than if they basically have to find that for themselves. Right, creating the strategy guide. They can't create the strategy guide for themselves, which is like pretty interesting. So like we do expect eventually they'll get better at this, but it's again, it's a big deal if that happens because now it means, okay, go back to that first benchmark suite. Okay, they're not great at making charts for us or coming up with data insights for us, you know, without guidance. but now put them in on the team for a little while and let them learn on the fly and maybe they do become good.

59:29Greg Burnham:So you sort of see these as a combination. Like if on our board game where it's easy to measure, they're not getting better over time, then it's okay for us over in the real world, like human in the loop evaluation studies for us to just do one try and say, okay, well, they weren't great and we don't think they learn on the, so that, you know, but really, you know, those go together. And is the board game a synthetic one that you created for this challenge? Or is it a real board game that people play? The real board game, it's called, or the first one we did this with, we'll do this with more, is called Earthborn Rangers.

1:00:04Greg Burnham:It's fairly obscure. If you go to like boardgamegeek.com, it's like 500th most popular or something. So it's not like up there. It's a small devoted community to it. And AI systems don't seem to know that much about it. Like they've heard of it. They can give you some facts about it, but it doesn't seem like something they've been trained on specifically. It's impossible for us to tell this with high certainty, with high confidence, but that's sort of why we chose it. It's just hard to make these things from scratch and know whether they're hard or whether they're broken. So that's why we, but the risk, which you might have in mind is, what if AI systems like, you know, just know about it from their training data.

1:00:45Greg Burnham:It's a risk on our mind. It's just a risk. One thing we will do in the future is we're moving on to video games. Incidentally, it's another area where AI is like very much not like human reaction time and visual and spatial reasoning. Again, coming along fast, but not really there yet. So video games, you can always test on a new game, like a game that just came out, not in the training data probably, probably, or at least, you know, not as robustly in the training data. And so we'll be testing, can AI systems do well on those? Do they do better on like, you know, the older version of a video game than like, you know, Grand Theft Auto 4 versus 5 or whatever, like when GTA 6, whatever it is, comes out, like, I don't think we'll be doing this one, but, you know, are they worse at that?

1:01:32It's interesting when you bring up video games, it immediately calls to mind kind of the classical reinforcement learning approaches, which is kind of learning on the fly, but kind of different.

1:01:44Greg Burnham:Yeah, we call the second sometimes in context learning, where the only like lever the AI system really has is managing its context. And like, look, when we give it the strategy guides, and they do much better, clearly, this is a powerful lever, but it's not as powerful as updating the weights. And like, so yeah, I mean, like, what's the big picture here? It's, it's, if you have this fast loop of in-context on-the-fly learning, then AI capabilities might sort of improve much more rapidly than the slower loop of collecting more training data, doing more reinforcement learning, which is done on language models as well these days, and then deploying a new model.

1:02:25Greg Burnham:Now, AI might automate that whole loop, but it'll still be slower than if it can just, you know, work out its own context on the fly, yeah. Yeah, it's interesting when you zoom out on some of the questions that you focus on, and we haven't talked explicitly about it, but you have a blog post where you kind of write out these nine big hairy questions that you're tracking. They kind of group into, like, how do I assess where AI is now? That's like, can I do my job? Like, how is it, you know, competitively? and, you know, where's the puck going? Like, can it learn on the fly? You know, can it do research?

1:03:07Can it come up with new ideas? These are all, it is kind of where it is now, but also like what, you know, is it positioned to like dramatically self-improve or like, you know, what's the trajectory?

1:03:22Greg Burnham:That's exactly right. The, I mean, I think for people thinking more broadly about what they need to know about AI. This is, the second topic is, you know, just as important as the first. But it's really, you know, even if you take some of the recent incidents, like the hugging face incident, like the sort of think of two axes, alignment of is the AI doing what I want it to do holistically? Obviously hacking into some other companies' computers is not well aligned. And then capabilities, just like, what can it get done? Can it hack into, you know, reasonably well-secured other companies' computers?

1:04:09Greg Burnham:What that incident was, was like an early example, very, like, I think pretty robust example of, well, we don't have the alignment thing solved, and capabilities are getting so big, so strong, that this is becoming a problem. And so paying attention to capabilities like what we should expect is becoming more important sort of day to day. And we're just trying to produce benchmarks that both say, where are we now? Exactly like you said, like, what can it do now? This is maybe relevant for more mundane economic utility or whatever, as well as, you know, do we see the dynamics of the system changing in a way that might be alarming, basically.

1:04:53I'm kind of also thinking about it as like first and second derivative of capability.

1:04:58Greg Burnham:Yeah, I think that's right. Some of these things make it the slope goes, you know, much faster, hyperbolic growth or whatever in some of these cases versus merely just like, yep, it's growing and it continues to grow. Both are important. I mean, I think we've seen the disruption we've seen to date has come from these kind of unpredictable thresholds. Like when, you know, GPT-2, GPT-3, suddenly you had the chat GPT moment and you're like, oh, this thing, I want to talk to this thing. I might use this thing instead of a search engine. This thing is useful to me for finding information. And then like a little while years later, whatever, two years later, you had the, it's getting better slowly and steadily at helping me with coding or with like using my computer, but suddenly - And thinking models.

1:05:47Greg Burnham:With the thinking models. And then you had the clawed code moment where it was like, oh, suddenly this thing can actually do software projects for me, even if I'm not an engineer myself. And it's hard to predict when these thresholds will be crossed. So it's useful to still track, but it's useful to track the mundane capabilities today and say, look, cyber is maybe the latest example, where it's like, look, we see their ability in controlled settings to do hacking going up smoothly. And at some point, it's going to cross some threshold where, oh, shoot, it can hack Hugging face or whatever. Hard to predict where those thresholds are, but still very useful to track those capabilities.

1:06:26Greg Burnham:So even the first derivative, as you said, version of this, we find very useful. We've got a fun benchmark we hope to release, maybe by the time this is published, we'll have it out on a furniture assembly where we build some IKEA furniture, make some mistakes and see if the AI system can like just given photos, like say, oh, like, wait, you made a mistake and step back in step four, like stop, go back. And they've, you know, we see the same smooth capabilities like Astra, GPT-6 Astra, pretty good at this, actually. And, you know, but it wasn't out of nowhere. It was like from a smooth and this is, you know, important because we haven't seen AI have much of an impact in physical industry yet.

1:07:09Greg Burnham:Like it's been mostly digital. and even before robotics hits the scene, we might expect that, you know, Claude in your glasses or something is saying like, hey, you know, I'm going to guide you through repairing your car or whatever. I'm going to help you fix this machine in the factory that broke. So, you know, we want to track those capabilities too, just for the mundane impact. Anyway, impact all over the place coming. Benchmarks help us say so. Awesome, awesome. Well, Greg, thanks so much for jumping on and sharing a bit about what you're working on. It's super interesting stuff and a very thought-provoking way of thinking about, you know, AI and where it is, where it's going.

1:07:48Greg Burnham:Thanks, Sam. Really appreciate it. Awesome. Thank you.

1:08:09you