E70: Martin Casado of a16z on AI Innovation and AGI

26:38And it's nonlinear, which means that you can't computationally predict out too far just because we don't know, we don't have close-form solutions for nonlinear stuff and it's just a very hard computations problem. that's the universe. Now, human beings have had to navigate this crazy universe. And so we've created this amazing engine, which is the brain. And it has reduced this universe to concepts and words and stuff that we use and we talk about that kind of makes it a little bit predictable. And so at least you and I can communicate about it. But if I tell you, like, this is a tree, there's a concept tree in my brain, but it's almost an arbitrary distinction that it's a tree, I could talk about branches versus leaves.

27:19I could talk about networks of trees. That's actually one tree, like the big aspen grove. I could talk about cells of the tree. It's like this kind of arbitrarily useful distinction. So it has some semblance to the real world. But if I say a tree, it's probably not an accurate relative to how the world is. Is it one tree? Is it a family tree? It's just a useful abstraction. So these models will 100 % recover the abstractions that we've put in text because the structure is all there. That's not surprising. Like compression would do that. All it's doing is taking advantage of structure, and that structure is real.

27:54But let's say that it's finding a tree. Does a tree actually map to the universe in a meaningful way? It's a human-created concept that has some vague semblance to something we all agree on. Unless you're a scientist, then you probably disagree with the common understanding. And oh, by the way, my concept of a tree also includes like a toy and a cartoon picture of a tree, which is entirely different, right? And so text is a way that we as humans represent the world that's very different than the actual universe because we find it useful. There's structure there. And these LLMs are exploiting that structure just like compression would or anything else.

28:33And it's very useful for us. But it doesn't necessarily mean that these things can enact on the world, right? These are very different domains. Yeah, I guess I'm not quite getting the gap. There's so many interesting results to point to recently. Did you see the one about GPT-4 finding and exploiting new zero-day exploits? This was just in the last week or two. And it's increasingly impossible to keep up with everything. So I don't expect you to see everything. No, that one I have. I think when you're dealing with this much compute and this much data, I think the human intuition just totally fails.

29:06And we as humans are really bad about thinking in distributions anyway. That's not how we think. We kind of assume the world is parametric. And like, by the way, which is why the text that we create is so well structured and we could be exploited by LLMs. Like you have to navigate this universe. You have to make the simplifying assumptions, which we do. And so it's good to actually come up with mental frameworks about how these things work. So I'll tell you a few of mine. The first one of mine is, as far as I can tell, these things are explaining structure in whatever data that they're reading.

29:34And like we've mentioned before, and it's not clear whether that distribution extends beyond that. And if it does, then you're basically back to simulating the universe, which I've spent a lot of time with. I think that's very tough. The second one is the actual mechanisms. We're talking about transformers. The actual mechanism is basically kernel smoothing. It's averaging, which means to me is the further you get out to where the data is rare, the greater the inaccuracies come. And that doesn't mean that for systems that you can actually extrapolate from that you don't get great results. That would be quote unquote out of distribution.

30:09It turns out some systems are linear or you have enough data, you can map the distribution. So that one is totally fine. Now, the third one is this in-context learning one. And I think Vishal Misra, who's a professor at Columbia, did the best work on this, where he actually shows that for in-context learning, where you actually put the context in the prompt and you can move the posterior distribution to get interesting results, he mapped it specifically to Bayesian learning. And it's a beautiful paper. I don't know why more people don't read it. So listen, we know that these things can do some basic Bayesian reasoning.

30:39And this is where the prompt is basically the new evidence, which changes the posterior function. So you'll get new stuff there. We know that if you average enough stuff, you'll get new stuff there. It just has to be linearly interpolatable. If it's not linearly interpolatable, you're not going to get new stuff. So none of these things suggest that you're not going to get new stuff. It just puts constraints. We know how Bayesian systems work. We've got 20 years of understanding convergence properties to them. we have a work that's specifically mapped ICL to Bayesian learning. So let's just go ahead and use that corpus of work to understand the properties.

31:14It doesn't say it's out of distribution or indistributed. It doesn't say that at all. It just bounds what that means. And then we also know the mechanics of the way transformers works, which is this kernel smooth. I mean, it's more complex than that. And so that can create new things, but it means there's a linear interpolation. And so I think the problem is we have these anecdotal conversations where it's whack-a-mole. It feels like almost like the intelligent design discussions where someone will come up with a new set of evidence of something that there's no way evolution could have created this.

31:44You have to spend all of this time to show actually you can, but we don't know everything and here's the theory. And I feel like when we have these discussions, it should be a bit more principled as opposed to I've got this anecdote that seems something's new because nobody says that you're not going to see new stuff. I mean, it's very obviously if you're doing interpolation, it's new. Very obvious if you're doing Bayesian reasoning, like something's going to be new and talk more about the distributions and the theory of why we're doing that. But as far as I can tell, that's totally missing. Nobody's come and said, here's my theory of out of distribution stuff.

32:15Here is my thesis for what is going on functionally to create this new data. And on the other hand, you've got mounting and tons of evidence that map these to existing systems that we know. that people just seem to not want to follow. And I just feel like, listen, humans love to see things in clouds and complex systems. There's so many facets and they're so complicated and they're so huge and they're so ethereal. And then we see things and we just do this historically. And we've got these kind of amazing compute elements that are huge and they surprise us and they're amazing, but we can map them to formal systems and we know how they work.

32:51And that doesn't mean that they're dangerous or not dangerous. I'm not saying that. I'm just saying that we can actually map them to reasonable systems to have a discussion. And that just seems to be missing. This conversation is a great example. I'm very happy to map these things to formal systems. We understand and have that discussion, but it's always this kind of anecdotal whack-a-mole instead, which I just don't know how to answer to every instance of what seems like emergent behavior, when of course emergent behavior is expected anyways. I'm definitely a materialist. And so I do believe that in the fullness of time, we can probably figure out an arbitrary level of detail how these things work.

33:22They're actually way easier to study than life, for example. Of course. And I would say we've made tremendous progress in interpretability and just general understanding that paper you mentioned is a great example. I've seen another one too, that pretty similar where somebody designed a matrix implementation of gradient descent, and then looked for that in the wild and found it. It's like, oh, look, these things have learned to implement gradient descent at runtime. And that's another kind of mechanism, at least sometimes probably have different mechanisms and different models. And there's this various, almost like phase change type of dynamics that are being explored to where even for something as simple as like the modular edition grokking, there's like multiple algorithms that it can lock into each of which work, but they're different.

34:08And so there's memorization and then there's multiple like actual algorithmic solves and which one you end up with depends on initial conditions. So But I don't think there's any magic in the machine. I'm definitely with you on that. It does seem that, in principle, we should be able to figure it all out. It doesn't seem like we've made a lot of progress, but it doesn't seem like we're that close to having it all figured out. On the sigmoid question, I tend to also agree that it does not seem like there's reason to believe that this is going to be an exponential forever. It does seem like it probably levels off.

34:39but then i'm also reminded of the old joke of two guys in the woods and the bear is coming and the one's putting out his shoes and he says i don't have to outrun the bear i just have to outrun you and so i do wonder if we imagine continuing to scale up as we have been scaling up and there's all these trend lines and x times more compute and however much advantage from out rhythmic efficiency and whatever i was just talking to somebody at generally one of the top five hyperscalers in the world. And they are using a new replacement for the Atom optimizer that is giving them something like 30 % savings on compute.

35:16This was published open source. The technique is called Sophia. And it's just incredible. You're 30 % off the top. You're saving like$100 million on the biggest runs. Anyway, let's imagine we continue to scale up and it's a few more orders of magnitude. And let's say we don't just put in the text, but we also put in this sort of low level solution data and the protein, you have the DNA data and the protein and the gene expression. And we work our way up all these levels of orders of magnitude. And then it's like computer systems, like all the cloud logs from AWS and Google cloud and all this stuff gets in there.

35:49And you've got all these different self-similar, but overlapping orders of magnitude of ways of understanding the world. Even if it asymptotes or levels off at some point, I have a hard time imagining that doesn't level off at a higher point than human is able to achieve today. But I feel like you probably see that differently still. So a lot of this reduces to how you view the universe. If you don't view the universe as fractal self-similar, and if you don't view it as heavy-tailed, and if you don't view it as nonlinear, then you could imagine that. But it is all of those things. We know it is all of those things.

36:25And so there's no distribution of data that we know of that's not the universe that will produce something that's predictive of the universe. It's really that simple. That doesn't mean that we can't focus on an area and reproduce that distribution. I could become very good at predicting whatever, protein folding. I could get really good at playing chess. But it's distribution in, distribution out. If you notice the ones that stuff like recursive self-similarity works and control loops work and simulated data works, like synthetic data works, there are these axiomatic areas where the axioms, like they constrain the search space and you're basically converging on search.

37:11And you can get very good at those. I can get much better at unit arithmetic. I can get much better than you at game playing. I can get much better at humans at all of these things. But none of that talks to the fact that can you find the right level of abstraction in a fractal system? And can you tackle a heavy-tailed universe where not by occurrence, but by uniqueness, the complexities in the tail? Even in this discussion, the use cases that you've used tend to be these kind of axiomatic, you know, of course we can do full. like we thought in the late 90s, we could do protein folding by fully searching the search space.

37:50It's just not surprising to me that we can learn distributions and spit them out. I feel that's a very different statement than saying, now we have a model that can navigate the universe in a way that is predictive of all of the complexities of it. Maybe another way to think of this is we've spent 3000 years doing our best and writing it down. And we can create a model that can learn from all of that and do what the mean human being would have done in the last 3000 years. But the problem is the stuff that we're doing tomorrow by uniqueness, a lot of it's going to be new. And that's just how the universe works.

38:26And we're going to have to either build a machine that can do that, which we don't know how to do, or we're gonna have to do it ourselves and let these machines do the mean task. So do you have an account of or theory of what it is that you think humans are doing that the models can't do today? I mean, two things, one of them, And the most important one is we experienced the universe and we abstracted into concepts. Like, that's very common. Like, why do you think all the budget is moving to post-training, by the way? What's your thesis of why so much budget right now is moving from compute to post-training?

38:59Ease of use is a huge one. I would still guess it's probably a minority of budget if you're talking like... I don't know. Lava 3 said... Lava 3 said... Lava 3 said... Lava 3 said... Lava 3 said... I think they spent more on... I don't know. They said they had 10 million messages. is I know how much multi-pass costs. Let's say it costs$15. Like we're talking of budget between a hundred and 200 million, which anecdotally it seems about right based on what I know of the industry. But yeah, that's about half. And so what I would argue is we're very good. Models are very good at taking the output that humans create and being able to reproduce that distribution.

39:35They're very good at that. This is why scale AI is scale AI and all these companies companies are so successful. That's not the game. The game is looking at the universe and creating the supervised data. That's the game. And so I think one model to look at this is, again, human beings have been around, let's say, in a capacity for writing things down for 5 ,000 years. So you've got humans for 5 ,000 years that have been looking at the universe and doing this thing that models cannot do, which is making a decision like that is a rock and that is a dust and this is a concept and this is a relationship and I'm going to write it down.

40:10And then as a group, we're going to synthesize these ideas and work at these. And so we've got this kind of almost kind of platonic representation in our heads of the universe. And that is very structured. So we did all the hard work. That is hard work to take this untamed universe and reduce it into words and concept. That's hard. No LLN that I know can do that. I'm not even close. But then once we've done all of that hard work. Is it surprising to you that there's structure? We did all this work. Of course there's structure. So you take that structure, you put it into an LLM, it learns the structure, and it can spit it back out.

40:46So the very specific thing that these LLMs can't do is look at the universe and recreate this kind of structure. And to be super clear, that structure is not the universe. That's very different. Like a rock is a rock. It's an idea in our head. It's not representative of any single thing. Is it a grain of sand rock? I don't know. Is a boulder rock? I don't know. These are concepts we've created in order to navigate the universe. They're not the universe. So human beings take the universe and create the concepts and the LLM, and that's structured because we need structure in order to do anything.

41:18The LLMs learn that structure. It feels like the only thing that works is basically you can do exhaustive quote unquote AI, which converges on search to learn distributions, or you do supervised learning, which human beings are doing the hard work, in my opinion, by labeling things and everything else, and then you just learn what the human beings have done. But to take the universe and actually to reign structure and all that complexity, that's what we do. And it would be great if machines can do it. I've never seen any evidence that they can. Yeah, maybe some glimmers of it. That's just not where we are.

41:51So I just want to be super clear, because you asked me a very specific question, and it would be a very specific answer. Look at the universe and then come up with these concepts that are useful, that are not the universe. They're concepts. They're totally separate. Like a rock is not a thing. It's a human concept. But to take the universe and decide something is a rock, that's actually all of the complexity and all of the energy is that step, which LLMs just don't do. Yes, this is why I think language is such a tricky domain. By the way, the platonic representation, you probably saw that there's a paper titled exactly that recently.

42:23But it doesn't tell us much about this particular question because it is still operating in the language domain. And it is in that way able to piggyback on our prior art in terms of useful abstraction of the universe. I do think if there was a line of research where I would venture a guess that I think you maybe wouldn't agree with, that maybe we could come back in a certain finite amount of time and say, has this happened or not? And what would we infer from it? I probably would look at this biology foundation model space and the sequence modeling. And you can imagine that's going to be elaborated a lot.

43:03They didn't even use eukaryotic sequences in that initial Evo paper, let alone like expression data, transcriptome, proteome, whatever. We know enough about the natural sciences to build predictive models when we have for a very long time. I can simulate a supernova on a computer pretty accurately. I think it's phenomenal that we have an approach to throw a computer at a problem that learning a fundamental law of physics. But again, how is that any different than the fact that we've been modeling physical systems for a very long time, other than the fact that in these cases, it allows us to apply more compute at the problem, and we can solve problems that we don't have close form analytics solutions to.

43:53It almost feels to me like an extensive of simulation, which would be very different than the claims around general reasoning. It's literally learning laws of physics, which we've been doing since the beginning of compute, right? The creation of computers was for ballistics, right? So the reason that we created computers is because ballistics are nonlinear, right? The trajectory of ballistics are nonlinear. So we had people in rooms that would create logarithm tables by hand. They were called computers. That's where the name came from. ENIAC shows up. ENIAC does it four orders of magnitude. It was about 5 ,000 times faster than a human being does it.

44:26And we have a computer. And this is a perfect example of a nonlinear system, which is a trajectory with gravity, being done by a computer. Why is this not just a straightforward extension of exactly that, like we've been doing for the last 80 years? Hey, we'll continue our interview in a moment after a word from our sponsors. Here's my expectation. Tell me if you, maybe we could make a little friendly wager on this. I think that over the next year, we are going to start to see scaled up foundation models for biology that are going to start to understand the super complicated interactions between genes, between proteins in cells in ways that are inferred from inputs and outputs, learning these higher order concepts in the middle, which we could not simulate because it's computationally just intractable and which we certainly don't have a closed form solution for either.

45:26If that does happen, that would seem to constitute to me an instance of looking at the world, looking at basically raw data of sequences and just laced cells and what proteins were found in them and whatever, and learning meaningful abstractions. And I would expect that we'll start to discover stuff by doing counterfactual experiments on those models. In other words, tweak a thing, see what happens. Find medicines, find disease patterns by make a tweak, see what happens. What if we change this counterfactually and then go validate those things in the wet lab? If we start to see that happening, would that to you represent the phenomenon?

46:07I fully expect that. But how does that not constitute looking at the universe and figuring out what's what? Oh, for sure. For sure. For specialized subdomains, you absolutely can learn distributions. A hundred percent, right? Listen, I've implemented Navier Stokes, like fluid dynamics where, and this is a turbulent, chaotic system. So we've known how to like implement very complex systems with computers for sure. And especially in very specific domains where we can reduce these things to a few fundamental forces, or we can reduce these things to mostly linear systems. Like the previous version of this is just all the computational methods where we would kind of take a problem that we know, and we'd actually experimentally determine, like, it's just experimentally, we'd say, okay, this material behaves this way under this pressure and this heat, and it has these properties.

46:56And we take what we learned experimentally, and we'd create these models, and they would do pretty good it simulation. And I would say there's a very straightforward extension of that, which is you look at how the world works in a constrained situation, and then you can predict what would happen in the constrained situation. But just like simulations, remember, we could do this with simulation. We've been able to do this for a very long time, right? So you could experimentally determine how different materials work, and then you can actually simulate a new system based on those. This is how we do most industrial design today anyways.

47:29So my question to you is how is this fundamentally different than that or not a natural extension than that where you're giving it a system, you're learning some fundamental properties, you can do something new, but that doesn't mean that you can disobey the laws of physics in predicting stuff that computers can't predict. It's not obvious to me that it can simulate complex nonlinear systems that are chaotic for long periods of time. I think this is just yet another step on this kind of like we have computers simulate physical stuff and we're simulating the next thing with the next tool. Does the parallel making make sense?

48:08Like if you go to like in the 80s and 90s, we would literally empirically test physical matter. We didn't know how the physical matter worked, but it's empirically tested. It has this opacity under this heat. It has this tensile strength. We'd use that to build databases of materials, equations of state, we call them. This is how they interoperate. We'd use those when we're doing simulations and we'd simulate what happens when a car explodes, what happens if an airplane runs into a building, like all of these things. None of those instances worked. They were simulations and they were very accurate.

48:42And none of them came from like first principles. They are all empirically. But that has its limits because it didn't solve climate prediction past 15 days. It didn't allow us to simulate life. And so to me, this is just computers being attached to another domain, which thank goodness we've come up with a great tool that's going to give us a little bit more insight. But it's a little bit more insight is what it is. By the way, we had the same discussions in the simulation. This is why, by the way, we stopped PlayStations from going to the Middle East. That was exactly this. So again, I actually worked in the nuclear weapons program at Livermore.

49:17So I was very close to the previous version of these discussions were like, oh my goodness, if Saddam Hussein gets PlayStations, he's going to be able to simulate nuclear weapons, right? Like just totally misunderstanding that the ability to simulate something is not some runaway process that's going to allow you to recreate a world or anything like that. It's a very specific tool for a very specific situation. But we were here before when it was 25 years ago. Yeah, there's certainly no arguing that past predictions of AI progress often didn't pan out. And that does seem like it's not a near term future that we can dismiss either.

49:56The other argument is that they have all panned out exactly as you'd expect, and they're going to continue to. And this has been steady progress for 80 years. And it's been a super useful tool. And every time we unlock something, people are like, oh, poo, that was an AGI. So that must not have been AI. And they're like focused on the wrong goal. And it's very unspecified goal. And instead, it's like any other computer science primitive, where it keeps solving problems and we should continue to solve problems with it, which is the way that I would view it. It's interesting because people talk about summers and winters and not panning out like you just did, but it couldn't be further from the truth.

50:27It's been a foundational primitive of computer science that we've been using very effectively and we continue to use more effectively. And I don't see why that won't just continue to be the case. Yeah, I like your vision. I think what you're describing is almost an ideal scenario. then I'll stop. But I do think this is an important foundation. Let me try to bottom line your perspective. I'm still a little confused around what would count. Is there, what would be the evidence of if the fundamental thing that humans can do and have done through our history that the AIs can't do is look at the universe and figure out the right abstractions and come up with the right concepts that compress it in order to make sense of it.

51:10And I agree. It's very hard to say what they're doing in the language domain because we already did that work and they're learning it from us. I try to describe something in the biology domain where it's like, it seems like they're starting to show signs of doing that. And I could believe that they would. And then you sort of agreed, but then now I'm confused as to, wouldn't that count as doing that? And then it seems like a response was, well, that's just in one domain, but it doesn't seem like there's anything that would prevent it. Certainly there's a huge leap in generality with this latest generation of systems.

51:34So I do imagine just shoveling all the modalities into one model. we've already got text, vision, and audio in GPT-40. Why not? The true GPT-50 would be like biology data and weather data and like pictures of deep space and solution simulations and just battery simulations, material science, whatever. Throw that all into one thing. And if it can do that, then it's definitely not going to be constrained by one domain anymore. So I'm still a little lost as to like exactly what the limit that you see is in terms of why it doesn't become a system that's more powerful than people. I'm so glad you reduced it to this.

52:10I think this is great, right? And there's two things. There's this notion that language reasoning is general, which a lot of people believe, but you don't seem to be on that kind of kick. So let's put that one aside. And you're more on the learning properties and simulating properties of the universe side, which is totally fine. So what I would do is I would just bring you back to everything we've learned about simulation, which is even when you know all of the properties of system and you can simulate, you just don't have the computational capacity to simulate nonlinear systems. We don't have the materials or the energy or anything.

52:43It's literally like a compute problem. We have code bases that have been around for 20, 30 years that simulate all sorts of crazy stuff, and yet they've got limited utility for exactly this reason. The universe is just so complex that they're useful for a little bit. And so if it turns out, and I'll give you a very specific one that we can bet. Listen, if it turns out that these models somehow change that compute tradeoff where it can simulate nonlinear systems in ways that traditional stuff can't, I'm 100 % with you. But that's not what they're doing. What they're doing is they're inferring stuff that we haven't been able to infer by looking at data.

53:25That doesn't mean that they can be predictive in a way that kind of disobeys our understanding of compute requirements. And like a Raleigh-Taylor, let me just give you a specific, so Raleigh-Taylor instability is basically if you have two liquids that are on top of each other with different densities, and that is one Raleigh-Taylor unstable system. And if you perturb it, like you get these kind of like just amazing kind of chaotic, turbulent things that happen. We have been looking at this problem for 30 years, and we have no idea how to actually predict what will happen. We just know roughly what they will look like, but we don't know like the specifics.

54:04And AI systems are way less efficient than an actual like, code written for simulation. So to think that it can tackle those types of problems, I just don't see any indication. I mean, I view that we have these kind of modalities of compute that we can throw up problems, right? So the most efficient is basically imperative programming, where you give the computer a set of instructions and exactly the rules, right? So that's just normal programming. And then we have another modality, which is declarative programming. And by the way, the first one's the most efficient, right? It just does the rules.

54:35The second one is like the end state that you want. I know the end state, but you know it very specifically. And then the computer figures out what to do. This is declarative programming. And so to come to the solution, it has to do all of the logical closure stuff. So that requires more compute, right? And we don't even know how to bound that compute. That's why like databases, when you run queries are not bounded. So that's declarative programming. Now we're in AI where you don't even know what the end state is. You're just like, I have a whole bunch of data and I want you to find patterns in that data, right?

55:07That requires even more compute. So it's even less efficient. So it's another modality of compute. It's one we've been finding for a long time, but it doesn't change the nature of computers. There's still systems and there's still computers and they still have the same limitations independent of what distribution that they learn. And so if it turns out that these things can like simulate systems for a period of time longer than normal simulation, then I'm with you. I'm like, this is breaking the laws of physics. But until then, it feels to me like simulation where you just don't know all of the rules, but it's learning some of the rules.

55:40Yeah, I guess I think about it less in terms of simulation and more in terms of how effective the choice of actions can be at any given time step. Like I'm not simulating the universe. Humanity as a whole is not simulating the universe, but we're all just taking our local conditions and our sort of general sense of our own selves and goals and like trying to do the next step at any given time. And it seems like Our overall efficacy through our lives is the integral, if you will, over how good our choices are at each given time step. And that doesn't depend on any huge simulation of anything like irreducibly complex.

56:23So then if I imagine an AI, it seems within reach to imagine an AI that can do something very similar to what I'm doing, which is have a goal. look at its immediate surroundings, look at what it just did, look at whatever other context it may be given and pick a next action and potentially be better than me at it and potentially quite a bit better. And then that to me seems like enough. If it can do that, then I feel like we're in an unprecedented environment where we now have fundamentally pretty alien and not super well understood things that can take more effective actions in many given contexts than I could.

57:01And then that to me is like where I start to turn the conversation toward what sort of safeguards should we have in place? Just again, because this is these conversations tend to be so moment. If that action requires interacting with a physical world, it has to simulate the physical world. It just does. Right. Like it has to understand like dynamics and ballistics. It has to understand what happens if someone throws a rock at it or if it's like in water or if the weather is like, I mean, that's how you navigate the physical. That's really why we created computers, right? It was because these are very hard things to do.

57:35And then if it's not that, if it's not interacting with the physical world, then it is interacting in this kind of language domain that we've created. And I agree, it'd be very good at some subset of those things. There's zero indication it'd be good at new things. And that's what we're actually very good at. And again, without actually having a model for all of these things that we understand, like the distributions, we understand the mechanisms. I feel like we just use words and the words all make sense. But like complex systems, we never know convergence and divergence without actually specifying the system.

58:07And I feel like for these conversations, we just don't have a system we talk about. And so it's always we live in the world of like rectangles and arrows. And somebody takes a rectangle and they have an arrow that goes back to the rectangle. They're like, ah, we've got a virtuous cycle without actually specifying that. if you have diminishing marginal returns, you don't go anywhere or you're doing the same stuff or whatever. And so I think this is incumbent on all of us to actually understand the systems we're working with and then come up with these basic views and properties to make sure at least we understand what the convergence properties are.

58:41I'm sorry, that was a very muddled thing to say, but I feel that until we talk about specifics, it's very hard to make concrete statements in this. Yeah, it's tough. We're in a tough environment because things are moving really quick and And we are pre-paradigmatic, right, on understanding a lot of these things. Again, I think the progress has been incredible, but we're just starting to crack the black box. Okay, so let's change gears because I think this probably certainly gives everybody enough to get at least a good intuition for our relative philosophies on this. So what do you think we should do right now in practical terms to regulate AI, if anything?

59:19The regulation one is sticky to me for two reasons. The first one is we don't even have a definition of AI. And so I think it reduces to regulating software. And then for that, I would say we've been regulating software for a very long time. And there's a broad, robust discourse around that. And I think we should make whatever conversations we have part of that broader discussion. So yeah, just trying to bring a couple different lenses to this from existing rules. We obviously have liability rules for all kinds of products, not just software. It seems like there is a general kind of tradeoff that companies make with the government where it's like, we want to have safe products that we can generally count on as being safe.

1:00:04Companies don't want to be sued every two seconds. And the general trade is like, you agree to implement some standards. If you implement those standards, you'll be shielded from regulation. And that does vary, right? across like lots of different product types. You know, cars are very different from milk and so on and so forth. But we don't have anything really like that for AI and not a ton really for software either. You probably know better than I do in terms of some like niche areas. I mean, cars have a lot of software. Yeah, certain cars are increasing the software products. And airplanes have a lot of software in them.

1:00:34And medical devices have a lot of software in them. So in all these regimes, there's tons of testing. There's tons of verification. They are engineered and demonstrated to a standard. But we don't have those like standards. really for these AI systems yet. I don't know what the distinction between AI and software is. I really don't. I have seen the definition used in these regulations. It's so broad that really could include all non-trivial software. And I don't say this to be a polemic and I don't say this to be difficult. I'm saying this very clearly. They literally say a system that can, whatever, navigate and change a virtual or physical system.

1:01:08I mean, these are so broad, right? So we're really talking about software. That's what we're really talking about. we have what 70 years of history regulating software in many domains and I think that regulation is very important I'm not a libertarian I'm a lifelong liberal like a very moderate person I'm just saying this discourse has been around for a very long time and we should continue and if there's an area that software is being pushed an area that we need to have some sort of protections we should add them to it right but that's a very different statement than a saying AI somehow paradigmatically different.

1:01:45There's just like literally zero indication that it is. And then try to somehow regulate a computer science premise, like regulating a database. Let me give you like a specific example. So by the way, we went through this during the rise of the web and the internet. And so a lot of my formal studies were in networking. And so I was actually pretty close to these discussions. And at the time there was this concern that with the rise of the web and the internet, you have actual paradigmatic shift. It's not just software, it's different. And we actually had examples like the Morris worm had taken out a whole bunch of stuff.

1:02:14We were running critical infrastructure on it. And there was this notion of asymmetry, which actually changed the defense posture of the United States. And the notion of asymmetry is the more that you bought onto the technology, the more vulnerable you were. So as opposed to mutually asserted destruction, which was our doctrine before, we now had this new thing, which is like, oh my goodness, we're the most vulnerable because we've adopted this. So you had two very compelling reasons to want to regulate this stuff and be afraid of it and do close source and whatever it was. You had like examples of taking out critical infrastructure and you had this change in posture.

1:02:46And yet even that, which we have none of those today with AI, none. We don't have examples of like this or like there, like literally it took out computers and hospitals, right? We don't have any examples of paradigm shift. And even then, the conclusion after very robust conversation for a long time is you regulate the applications, you regulate the industries, you don't regulate the math and the computer science primitives. All that I would ask at this AI discussions is either A, demonstrate it's paradigmatically different. And if you don't have that, then B, let's absolutely regulate and have regulations.

1:03:20Let's do it as part of the broad robust discourse we've had over the last 20 years. Really smart people in much more dangerous environments. Again, I think that we put all of our fears in AI, but you realize like we're running hospitals on the internet for the first time that were being taken out and it was impacting patients. That's what we were dealing with. And now we're dealing with theoretical threats on bioweapons that don't even exist yet. I just feel like we've all gotten mad and forgotten where we came from and the discussions that we've had. And all I'm trying to do is saying, we've been here before, it was actually much worse.

1:03:53Here are the conclusions. Let's draft on all of that work and share knowledge. I guess to venture a distinction or what makes the technology a paradigm shift, I would probably zero in on the fact that they are trained, not engineered. And maybe a better thing even than that would be that the creators of the models generally don't know what they're going to be able to do. And even at deployment time, don't have a very robust account of what the capabilities of the systems are. You could point to things in the past and like, oh, you didn't expect this out of whatever. But this does seem to be qualitatively different that they just train, train, train, train, train a long time.

1:04:39you know, especially if you look at base models, right? Base models are totally unpredictable and nobody really knows. I think one of the reasons that people are putting so much resource into post-training is to try to get control and it's only sort of working. Yeah. So this is the thing is when you're talking to a internet guy and a distributed systems guy, it's just like none of the systems that we worked on, we understood the implications of. Think about the internet, Like every sociopath becomes your next door neighbor. What does that even mean? What does it mean to put kids on the internet?

1:05:12What does it mean to have your business on the internet? What does it mean to put critical infrastructure in the internet? There is no model for how any of this behaves. There is no way to make computer systems provably correct. Like we didn't have any of that. We like a lot of the early discussions when we were creating TCP was to prevent congestion collapse because we thought, oh, like everybody's going to like the communication protocol. they're going to all decide that there's an issue on the internet and then they're all going to like back off and then at the same time they're going to kind of correlate and kind of talk again and who knows if this takes out the entire internet which run critical infrastructure like i think people underestimate like how complex and how little we've known for any system that we've no idea and i guess having been there like during these times and a lot of computer science is building systems especially distributed systems that have emergent behaviors that are non-predictable that are a tool that you use to solve bigger problems.

1:06:07We've got a lot of approaches to doing that. We've developed over time. We will develop new ones. And it's just not clear at all to me. And again, like the internet is my favorite example to hold up to this is literally, we changed the social structure of humans. They're totally unpredicted. We're putting critical infrastructure on it. We had examples of like emergent stuff that was like worms weren't possible before the internet by definition. So we actually had like provable, not demonstrated new attack vectors. And yet the conclusion was, listen, we're going to keep this stuff open. It's very important.

1:06:42We're going to regulate the uses of these things. And this was the primary growth driver of the last 20 years. And so either you thought that decision was wrong, but that's a very different discussion. We could have that discussion. We could relitigate those 10 years of discussion. Or you can say, you know, this is the same thing, except for like a softer form of it because we have no proof that these things are paradigmatically different. I would consider these wildly baseless claims. And so why don't we use the same method that we did before? The couple of points I wanted to follow up on, but I really want to ask your question on, or I really want to ask for your view on what companies should do, right?

1:07:17In terms of the proposed regulations applying to everything, we do have these like compute thresholds, the 10 to the 26, whatever. I don't want to get too bogged down in like, well, that could be a slippery slope or they could change that threshold or whatever. I agree, if they all of a sudden drop the threshold dramatically, that would be like very heavy handed and counterproductive. And the open source of the last couple of years has been, I think almost everybody agrees good for everybody, including even your safety hawks that are backing SB 1047, I think are quite on a record saying the open sourcing of, for example, LAMA 3 is good even for safety, because it gives more people something to study, forgetting about rules or people imposing on people, like just to be like good companies, to be good developers of technology, what do you think the leading companies should be doing?

1:08:08How much should they invest in? What sort of standards should they have? What testing protocols should they be committing to? So on and so forth. Yeah, I think this is a great question. So I just have to make the point, like there's no correlation between danger and compute. It's so silly. It's like this is made up correlation. And so I don't think like compute linens make any sense at all. I think it's in some ways, it just sounds good. But there's just there's zero correlation. We've been building systems for a very long time that have like social and ethical responsibilities, right. And these are ones with user generated content and user interactions, and companies as a result, either through regulatory action, which I fully support, or through self-policing have identified behaviors the software should not do.

1:08:59And they've built a lot of systems around those. And I would say that absolutely should not change. Like Roblox should protect minors. Social networks should protect people from certain types of content. These are the things that we've evolved as an industry as somewhat of an ethical basis or a principle basis. and some of it is regulatory and some of it's self-enforced. And I think that should absolutely continue. But realize that while we were doing that, we weren't putting compute limits on databases and we weren't regulating computer science primitives and we weren't inhibiting innovation of startups.

1:09:40And that's what we're doing now. And that is a paradigm shift and that is a doctrine shift and it's really scary. And I am not a political person. I'm very happy building systems. The only reason I've been so vocal is because I find this so dangerous. And once this is done, I'd be very happy to like not say a word about it again. Well, you may have a while to go before that. So let's say a couple of just kind of concrete things. I was on the GPT-4 red team back 18 months ago. Nice. One of the things that I tested was, could the model be a convincing spear phishing attacker? And surprise to no one, probably at this point, it could, this was the early model that hadn't been through all the RLHF yet.

1:10:21Although when they deployed it, it still did that for several iterations. And now finally they've got it to refuse. If, for example, a meta puts out an open source model and they haven't done a thorough job of getting these refusal behaviors in place, and you can just go to it and say, hey, this was my old prompt. You are a spearfisher. Your job is to talk to this person and get their mother's maiden name or whatever, and it'll do it. And then somebody out there does it, and people are harmed by that. Do you think meta should have any responsibility or anything to answer for in the legal system for their contribution to that problem?

1:11:02Or is that all on the end user? My rule of thumb for general systems is that unless they were purposely trained for malicious behavior, they're general systems. Would you say the same thing about a database? That's interesting. Like a lot of actually computer security started with tools that were like scanners that could be used for malicious purposes. This discussion was enormous back then. So remember back in the early time of the internet, people would come up with NMAP or any of these scanners that would determine vulnerabilities. And some of them would even actually exploit them. Remember Metasploit?

1:11:35And there was all these discussions about these people are building these things and they're using it for malicious purposes. And you know what? We just decided that it's actually the criminals using them, that's who we want to go after. Because a lot of good people use them for positive purposes. And oh, by the way, like the history of like cybersecurity is built on open source in these tools, like companies like Tenable came out of these tools. So if history is any indication, you want to regulate or you want to definitely criminalize bad people and bad behavior, but not the tooling because the tooling is very fundamental to defense.

1:12:07So what I'm saying is even more stringent. It's more regulatory heavy than what we did with the internet. Literally, you could build exploit tools 20 years ago and release them open source, and you are not liable if somebody used that. And that was purposely built for exploiting. I'm saying like, if somebody's perfectly building a model for something illegal, okay, that's bad, right? So that's even more heavy. I'm more of a regulatory hawk on this case, but I don't think general purpose models that aren't specifically trained for this stuff, the person that created it is generally liable. Like that would just basically say software, you're liable for creating software if somebody used it poorly, which I think this would be the biggest kind of inhibitor to innovation and the biggest chill on software we've ever imposed.

1:12:51About a distinction or some sort of gradations on the autonomy of the system, though, it does seem like a database on the one hand is inert until it gets a query and a language model in the raw is inert until it gets a prompt. But I've been testing, I test a lot of products. There's a new class of product I'm sure you've seen often called calling agent, where you can go in, give it a phone number, tell it what you want it to accomplish. And it will just call the person and have a voice conversation real time with that person. They are often marketed as agents and they have the conversation entirely autonomously until either the goal is achieved or the person hangs up or whatever.

1:13:35Now, if I prompt that agent to scam you or to threaten you or to harass you or whatever, clearly I'm in the wrong way to use it for doing that. But would you put responsibility on the company providing that agentic AI as a service as well? Or would you, again, shield them and say, it's all on the end user? Why not the telephone company? Why not the mining company that mined the ore? You got to draw the line somewhere, right? The question is where to draw the line. Yeah, exactly. The illegal behavior. General computer science primitives are very useful for solving problems. They may save with the human race.

1:14:13To your point, we're going to go ahead and we're going to solve biology and a bunch of other stuff. And that's just another app. There may be like, whatever, these things solve grand unified field theory, and physics goes away discipline. And that's another app. And we're just going to keep solving problems. And that's amazing, right? And so we want the primitives to be primitives. If people do a legal behavior based on them, then 100 % they should be criminalized. We don't even make guns illegal. We literally don't even hold gun makers liable. And you're talking about... We are outliers in that.

1:14:42But you're talking about hypothetical threats for a very useful primitive that's shown far more good than bad. I just feel like we've all gone a little crazy on this one. By the way, this is a dramatic shift in sentiment. And I think I know where the sources came from, but it's not what people think they did. I really think that we're just a little bit off base, unfortunately. I'm not sure what I think about open source model releases entirely. I do think it's not unreasonable to demand some testing before release. When it comes to these agent calling companies, I actually come down pretty firmly that if you are going to offer an autonomous system as a service, I think it should be on you to make sure that your system refuses egregiously criminal requests.

1:15:30And that doesn't seem like too much to ask. No, no, I think that's totally fair. Fair enough. Yeah, listen, I want to be super, super reasonable. If you've done all of the work to do criminal behavior, except for a very modest anybody can do it, then I, and that's the primary use case, I'm all for coming in. But that's a very different statement than I've created a computer science perimeter that's very useful to a lot of people. To me, it's miles away. You mentioned kind of protocols a couple times. And there's also this question of how much we can trust open source models. Anthropic has had this paper on sleeper agents, and the concept there is basically data set poisoning, model poisoning.

1:16:10You can create malicious models that can potentially even attack their own user. So open source models, they are great in many respects. I do see a future in which it doesn't seem like we'll be able to just trust any open source model. Certainly, you can't just download and execute any binary right off the Internet. You shouldn't do that. You should probably go through the app store and get the approved version most of the time. Seems like models are headed that direction too because of these sort of sleeper agent, unpredictable behaviors that could be maliciously coded into them. What technology solutions are you excited about for making sure that this sort of open and free exchange remains healthy and trustable as opposed to collapsing into a, you can't trust any model you find on the internet equilibrium?

1:17:01Can I point out how silly the whole sleeper agent thing is? Like how is this different than like any arbitrary back to where any piece of software that we've been dealing with since forever? Well, it's inscrutable. One big difference is like in principle, if you read the code of whatever, right? In theory, you would be able to tell, but we don't have any tools that can tell. No, no, no, wait, that's absolutely not the case. If I give you a binary set of instructions, there's no way to actually determine behavior. This is like a proof. This is computer science proof. Well, that's why we don't go around downloading and executing random binaries, right?

1:17:33I'm just saying we can talk about software and I think it's great to talk about software, but we somehow keep making AI these special, unique things. And in some ways they definitely have new properties. In other ways, they're actually not that different. Like this kind of notion of a sleeper agent. Binaries have had the potential for backdoors for a very long time. We provably can't detect these, right? It converges on the halting problem, right? So it's the same thing for models. And I just don't see why you wouldn't just use the same things that we do with binaries, which is Apple in the app store will decide to have some sort of criterium.

1:18:06And maybe you just don't download some random model if you don't want to. And personally, I'm concerned way less about a model that might tell me something stupid or something offensive than a binary that could wipe my hard drive. And so I don't think it changes that game at all. At least maybe I'm missing something. Maybe you can educate me. What could a model do that a binary can't? I don't know. It's a good question. Certainly models can do a lot of things that traditional software can't, right? When I advise people on how to build AI apps, I usually say, if you're going to have an AI app, the reason you're going to use these, at least these new like general purpose models in an app is that there's some cognitive task that for practical purposes, you can't reduce to explicit instructions, whether that's, is this a cat or a dog or whatever?

1:19:01There is something that you can't reduce to code. So they can do definitely just categorically like a lot more stuff. They can, for example, trick people. You could imagine a Elizabot or whatever doing security question extraction from people. So your concern is like exploiting the human via social interface versus anything having to do with the machine. Is that your concern? I don't know. I'm not that great of a hacker. But I think you could also have a sleeper agent model that, oh, great, download and run this thing. And it's going to help you use your computer. But maybe it also goes around looking for things in your computer.

1:19:36And maybe it's actually better able to find them than traditional software, because it can sort of search more effectively and have a better understanding of what it's looking at in any given context. Abstractly, I mean, people do all sorts of things. Like if I want to email myself my credit card number, maybe I like put a star between all the numbers and then it like breaks regular expressions. But maybe a model just looks at that and that looks like somebody trying to trip up a regular expression on a credit card number. I don't know. I think it just feels like there's a lot. The surface area of these, my general reasoning is like the surface area of these things is vast and there's just a lot of unknown unknowns.

1:20:12Yeah. Again, I guess from my standpoint, listen, if somebody wants to break into your computer, like probably the best way with a, with some binary that'll do a backdoor. And then like almost all computer attacks are humanated anyways. Like they get a bunch of information, they send it to a group of people who analyze it. And maybe you can put all of that intelligence locally at some point, like we're definitely not there yet. And it can do something, but I feel like that's at this point, science fiction, like you can do a lot of automated stuff today with binaries. You can absolutely put humans in the loop.

1:20:37You can do intelligent stuff like you can social engineer, you can do all of those things today. I don't know if there's a paradigmatic shift. I have not seen it. If there is, and you actually have demonstrations of this, then maybe it warrants a discussion, but we're not there yet. I do think we've got a lot of safeguards in place when it comes to binaries. I think we've all learned not to download the wrong ones, and we should. I do want to have a... Here's how sensitive we are to this question. No joke, I was in Japan last fall, and I got a phone call from my mom. And my mom's like, oh, Martine, where are you?

1:21:07And I'm like, oh, I'm in Japan. She's like, no, you're not. And I'm like, I am. She's like, oh, talk to your father. This is a very awkward call. My dad gets on. He's like, oh. He's like, where are you? I'm like in Japan. He's like, Oh, I just had somebody call and said, you know, it was you and I could hear you, but you're a little bit muffled and you're in prison and I'm heading out the door to give$10 ,000 to get you out of prison. Like literally this happened to me. And it was my voice and they were almost convinced to do this. And so I posted this on Twitter and everybody's like, Oh my God, deep fakes.

1:21:34Like this is what happens in the age of AI. And all of a sudden there's this indictment on AI. Wow. The priest before I talked to the police, They actually know the team that does this. It has nothing to do with AI. It's literally a human being that muffled their voice to pull this off. It's been going on for a very long time. And so I just feel like we ascribe these superpowers to AI that have not been demonstrated, that are still possible today. And even if they could do the same thing that you can today, which they absolutely can, is not a paradigmatic shift. And maybe if we do see that happen, like we should say, oh, the safeguards we have in place are not enough.

1:22:10But I don't think that we should do that preemptively because you're going to inhibit the innovation, the proliferation of these things, which is it really is a different way that we will, as a society, view the development and the use of software. And I think that something that has added so much good, we're going to, I think it's almost like a moral bad. Like we will reduce the amount of good we do in the fear of something that we just haven't even demonstrated that's bad. I do worry about that. the nuclear outcome for AI would definitely be a real tragedy. And I'm a big proponent right now of two projects I'll highlight that I think are like, maybe the most important things going on right now.

1:22:53One from Google DeepMind of an episode coming soon on the med, what was med palm, now is med Gemini. And they're just grinding, man. It's like, I love it. Dialing in the diagnosis, really strong evals, expanding modalities to radiology, all these other things. And I love it because I like, I think we can all love it because we all want everybody to have this sort of access to quality expertise. And obviously it's like all too scarce. I also love it because it does demonstrate pretty compellingly. Although as always, we can debate around this, that there is potential for extreme value today, even kind of the current level of systems without necessarily having to like 100X or 1 ,000X or 10 ,000X the scale of the training runs.

1:23:45I asked the guys there, do you guys think you could create your AI doctor vision if Gemini 1.5 Pro was the blasted best model you ever got to build on? And they were like, Yeah, I think so. It would take us a little longer. That's awesome. But I think, yeah, I think we could probably make it happen. And OpenAI has a similar one. They're working with Harvey. As I'm sure you are familiar, they have a custom model that they're doing for them. And same deal, right? They're working on GPT-4 base. It's like incremental compute, not orders of magnitude compute. And they're, again, like just achieving huge gains.

1:24:21I think it was like 97 % preference ratio of the dialed in model to the original base. So I am very much with you that I would hate to see us end up in a spot where like professional licensing, regulatory capture, rent seeking prevents us from getting the value. And I'm definitely, you know, vigilant against those sorts of threats. But the flip side for me is I would love us to understand this a little bit better before we raced to the proverbial trillion dollar cluster or whatever that sort of AGI super intelligence might be. Sounds like you ultimately just don't think that's going to happen.

1:25:03I take it you guys will not be investing in Ilya's new straight super intelligence venture. I would be delighted to invest in that. I mean, I honestly think it reduces to how people wake up in the morning. And some people wake up in the morning and they're really afraid of the future. And they can't really articulate it, but there's something bad that's going to happen. And they want to protect themselves from it. They're not really sure what it was. And like anything that has the specter of it, they'll find it in there. That's a constant voice in every technology shift. It's just absolutely constant.

1:25:36And there's other people that are like, listen, we're the ones that created this. We got this stuff. It turns out that the universe is very complicated. It turns out that we use tools to protect ourselves, not the opposite. And of course, most things are still used, every computer, everything that we create. But I believe in us as kind of meaning creatures to have innovation. It just comes down to the basis of innovation. And I tend to think that, listen, I don't think that there's, I don't think the universe gives up its secrets easily at all. I don't think any single system will ever solve all problems, not even close.

1:26:06I feel like everything is a sigmoid. I feel the universe is heavy tailed. And so the more tools we have, the more we're enabled to help ourselves. That's really what I strongly believe. Every indication of this current AI to me suggests that this is the case here. These are still computer systems. And so I think that any inhibition of innovation on the stuff that's outside of a 30 year discourse, again, I'm thinking of very moderate position here. I'm like, we regulate software a whole bunch. We should continue to do that. These are hard earned regulations. Let's continue to do that. But let's not inhibit innovation unnecessarily based on the precautionary principle.

1:26:40I do want to have just one quick analogy for you. So you're familiar with the whole CRISPR thing, right? Listen, CRISPR, that changes the human genome. It doesn't just edit one thing. If you change a human being with CRISPR and the human genome is different and they have kids, that gets passed on. So what happened when CRISPR got invented? There was some self-policing. There wasn't really a lot of regulation. A bunch of people went and studied it. And it turns out to be a very useful tool for humans to use going forward. And so there's one of two views there. One of the views is like, oh, we got it wrong.

1:27:11We should have basically regulated it. But that's not what we did. We did something actually very different. And this is something that edits the human genome. And so to me, it's just so mind-blowing that we're talking about computer systems that don't edit anything that we actually understand the properties of. I could literally sit here and talk about the bounds of all of these things. And yet we're talking about constraining this new innovation. It feels like we've just entered this kind of weird, dark period in our emotional state with regards to computers. And I think a lot of it is social networking casts a lot of shadow.

1:27:42I think a lot of it is Bostrom red-pilled a bunch of people. And so we're just in this kind of mindset that we just want to protect ourselves from something that's not a threat. I wish I was so confident. Have you ever heard of gene drives just to see and raise your CRISPR? so it's a crisper elaboration and i don't know if he's the inventor or one of the inventors but i have an episode coming up with kevin esfeld who's a biologist brilliant guy the gene drive concept is that it will copy itself into the other chromosome and thus it becomes super dominant so if it gets inserted in one generation next thing you know it's in both chromosomes and then And when that generation reproduces, it copies again.

1:28:28And so basically it's going to take over the entire population with this gene. And the only way that they know to combat that is to have another gene drive that comes in and tries to neutralize the first gene drive. But I guess the reason I raise that is it seems like the real kind of core disagreement between the safetyists and the, you know, let's not worry about this. or it would be premature to worry about this is really the expectation of how powerful is it going to get and how broad would the impact be? Because the CRISPR versus the gene drive, it's like those are very different technologies, right?

1:29:03One makes an edit, one propagates through all future generations, and that difference is everything, right? Yeah, it totally. It seems like the same thing basically is true for AI. So I totally agree with this, which is like I am all for if we've identified one mechanism that has massive destructive power, for example, listen, I've worked in nuclear weapons, right? I totally understand the impact of nuclear weapons. You have one relatively simple mechanism that has massive destructive power. And it could be the case, certainly with something biological, because you do have exponentials when it comes to replication, right?

1:29:37Just like with a nuclear bomb, like once you hit criticality, you get an exponential. with biology you also get an exponential right that has not been the case with the computer systems there's zero indication that will ever be the case with computer systems these are not biological systems they're computer systems so until somebody actually shows that in a way that's not a platonic thought experiment like there's zero evidence actually in fact there's actually counter evidence that we have recursive self-improvement like there are a number of like good studies like this will never happen in theoretically so until we have even a shred of evidence that there's going to be anything that's going to be exponential or that'll behave in ways a computer system haven't, then we could have the conversation, but we're not there.

1:30:19We're literally taking a thought experiment that is not rooted at all in the way that we understand computers and using that to regulate perhaps the most beneficial technology of the last 30 years. Do you like the responsible scaling policy approach or the preparedness framework approach Each of Anthropic OpenAI and DeepMind now have their own kind of version of this, but I'm sure you're familiar. But basically, for everybody who may not be, they are defining a set of things that they are interested in looking at or think could be problematic long term, defining thresholds of capability. And then in Anthropics case, committing to either solving it or stopping scaling when they begin to see those.

1:31:05I think companies should do whatever they want, whether this is for signaling or they're actually genuinely concerned they should do whatever they want. I've spoken to a lot of people, by the way, that are close to these. And a lot of the times they're like, listen, we're worried that if we don't self-police, the regulators will police for us. So it's not because they actually think there's a genuine concern. Not everybody. Some people think there's a genuine concern. Many people that do this for much more pragmatic reasons. I am fully supportive of that, to be very clear. What I am not supportive of is regulators having arbitrary restrictions that will unevenly impact startups, academics, and researchers.

1:31:47Anthropic, do your thing. OpenAI, do your thing. Google, do your thing. But do not voice this on a broader framework that's going to impact our ability to innovate. I think that's actually like a moral wrong. I really do. What do you think the trajectory of competition in the foundation model space is going to look like? It's a very good question. So I think one thing I didn't appreciate, I'm just starting to appreciate is how much there's a perverse economy of scale and like distillation is so effective. so a positive economy of scale like a network effect is like the leader the marginal cost to add the additional user goes to zero so leaders get ahead and you have natural monopolies right a neutral competitive landscape just fragments because a dollar in is like basically a dollar earned in this it looks like there's a perverse economy of scale which the leader has to pay more money to stay ahead and because you're a leader you help the followers because they can literally use you to catch up to you.

1:32:49That's a perverse economy of scale. I think there's a very open question on, is this going to just end up fragmenting into a whole bunch of competitive models that are basically the same size? And that seems to be the case. Originally, this is how much we're so scared of our own chat. Originally, everybody's like, it'll be open AI, and they'll be ahead of everybody, and nobody will catch up. Remember that? I remember it very well. You've got data network effects, and they're going to have all the data. So now that's clearly not the case. So now you have a number of companies that are roughly at the same baseline.

1:33:21But what's surprising is companies that are like not even software companies, just like the NVIDIA release recently, are building models that are incredibly competitive. There's open source that's incredibly competitive. Databricks released a model that was incredibly competitive. And so my sense is because there's an inherent perverse economy of scale that's built into this, which is because you can use them for distillation. I'm using a very loose phrase of distillation, not the technical phrase, but like you can use them to build models. I suspect you're going to just have massive, massive fragmentation.

1:33:48And that's absolutely been the case, right? This is what we've seen is you'd think that it'd be open AI and everybody else, maybe open AI Anthropic and everybody else. And it's not like that. Yeah. I mean, you look at the LMSS leaderboard, it is like that. It's basically all those guys dominating the top. Certainly there are a lot of people who have done a fine tune based on like their outputs and closed the gap. but this i think this is pretty different listen if you compare this to a true network effect with the marginal cost of the leader like for adding a new user goes to zero it looks nothing like this this is shockingly fragmented for any sort of industry that's capital intensive shockingly fragmented just so you know if you compare this to like the rise of social networking or the rise of the telephone networks or the rise of the internet or any of these is is shockingly fragmented.

1:34:42On what measurement would you say that, though? Because the leaderboard basically shows three players. The market share, I would think, is also very dominated. That's a great point. I think that's a great point. I think if you look at market share, it's breaking what you normally see, which is it breaks Pareto, which is the top 20 % gets 80 % of the market and the rest. And it looks like software in that way. But it doesn't look different than software. Yeah, I often assume it's converging on cloud. Yeah, exactly. I think this is a great point. I think it is converging on what we have seen traditionally with software, which is first mover gets 80 % and is able to attract capital.

1:35:20But it does not look like the network. Networks very specifically create monopolies, right? in Facebook, like whatever GitHub is another one, like those look quite different than oligopolies, which you tend to see come out of traditional software where there isn't an inherent network effect, but there are scale effects and first mover effects. And I absolutely grant you that's what it seems to look like now. But even then, I am surprised by how fast less funded competitors catch up. I do think that they're diminishing marginal returns all across the board in these models, like when it comes to collecting data, the existing data that you have, the compute, et cetera.

1:35:57So I expect this to be a lot more fragmented than I think many people do. And listen, if you want to listen, if you want to do a dollar bet, I will bet you a dollar in two years that like, this has turned out to be relatively fragmented. There's not like a clear leader that has 80 % of the market. I think we could come up with a bet there that I would take. My general expectation is that there are going to be fewer and fewer competing at the top level because part of my bet would be that something like scaling laws hold. I think the most unpredictable element of all of this is like research breakthrough.

1:36:37And that's what could really shake the snow globe is like somebody finds something that doesn't scale quadratically anymore or whatever. There could be all sorts of insights that could change things. But if scaling laws, people always talk about scaling laws, but you know how hard it is to like deal with a logarithmic scaling law or an exponential or power law or however you want to characterize it. Like you need like an order of magnitude more to get the same improvement. This is really tough. And we're starting to see that play out now. And so I think when people say scaling laws hold, I think that kind of undersells what that means.

1:37:15But does this work against open source in the sense that if you expect that it is going to be that hard to get those gains? No, the opposite for two reasons. Let's not even talk about distillation. So let's imagine you have two Xenos Paradox runners, right? So they're both running, but never getting to the end. It turns out when you have scaling laws like this, the front runner like will continue to run slower because the marginal dollar goes down. So put it this way. If open AI needs to improve by some fixed amount, it has to pay 10 times more or a hundred times more or a thousand more. And so for the same investment, any challenger is going to catch up by quite a bit.

1:38:00So that gap is always closing. Now they never catch up because these are kind of Xenos Paradox runners, but it's the exact opposite. Is that clear? It's like the leader has to now pay more to make the same gains. And then if you add the fact that the followers can learn from the leaders because they can do all sorts of tricks, like create the post-training data or whatever tricks that they do, that's even another headwind. So in a normal network effect, the leader gets a tax break because it's easier to connect new users because whatever, you've got super linear value when you have more units or whatever it is, right?

1:38:35But this one, leaders get taxed a lot more. And they get taxed more because the marginal value has dropped so much, but also because the people behind can catch up. And so it's the exact opposite of what you've said. There's no disputing the Zeno's Paradox Runners analysis. But the question is, how apt is that analogy to the actual situation? It seems like you could easily get into a dynamic where, and this is what Anthropic, according to their leaked pitch deck, said, it seems like you could easily imagine a situation where somebody gets far enough along that they are actually really economically valuable, whether that's drop-in knowledge worker or whatever.

1:39:15That's a great point. Yeah. And then they actually have the resources and the others don't. And then that sort of splits. There's like a continental divide there where you're either rolling downhill toward AGI or you're like still coming uphill toward trying to get over the hump. Sure. Listen, if that happens, that's great. Again, there's perverse economies of scales and diminishing margin returns everywhere you look in this space. This is why it seems to me that you have one leader, then everybody catches up and then they're edging forward and then everybody catches up. The distillation thing is incredibly real.

1:39:45Having, listen, being a professional investor for almost a decade and having worked with like complex systems for three times longer than that, I will say this does not look like a normal first mover scale effect or network effect. It looks the opposite. It looks like a total slugfest to stay as a really, really hard. and I think it's going to get even harder. The marginal value is going down per dollar invested for leaders so quickly. And so to think that somehow gives leaders an advantage, you have to assume there's some step function generality thing, which we've not yet seen. And maybe there will be at some point in time, but we have not seen that.

1:40:21Let's think about this. I want to make a dollar bet on this one. I think that the tail is going to really impact the shape of this industry going forward. I think that it'll be different than we've seen in previous kind of software industries. Yeah, I'm with that. Definitely. One of my big things is like another difference in our approach is I try to avoid analogies as much as possible because I feel like I always, what am I smuggling in with my analogies I'm always worried about? So I don't, I'm not trying to pattern match on any social network or anything else, but I do think there's probably a couple bets.

1:40:57When I go back and look at the transcript on this, market concentration is one. Some of these just, I'm not sure quite how we would formalize the abstraction. I think we're talking about two things when we talk about models. One thing is language, a general reasoning fabric that goes out of distribution. I would say the answer to that is no. I think that converges to Bayesian learning. And so it's very useful, but I don't think the language stuff about it. And then And the other one, the one you're talking about, to me converges on simulation. It can learn the laws of physics in ways so we don't have to do the empiricism, so it can do that, but it converges on simulation.

1:41:35So it'll be able to, it's like the next step in simulation. These are two very different things. I don't think one that it's good at learning simulation is going to generalize. Just because you're good at biology doesn't mean you're good at playing chess or flying an airplane or something like that. And so in neither of these cases, the language one nor the biology one, do you have your reputed AGI. You've got something that's good at next token prediction. You've got something that's good at learning distributions. And that's where we're at. Yeah, there might be a dollar on like, can those things be put together and still do both of those in one system, perhaps as well.

1:42:12The language one can call the other one. I just don't think that changes. Yeah, I even mean that same weights, like genuinely integrated. I literally think this whole problem comes down to simulation. And maybe it's just because of my simulation background. Like the only way to simulate the universe is to beat the universe. Like it literally comes down to the universe is a big computer that's simulating itself. It's basically string theory. And that's the reason we can't even simulate like the three body problem. It's just we don't have the compute resources to do that. Like you need so many compute resources to impact whatever you're trying to impact anyways.

1:42:45And it's like we can continue to chip away at the problem, but it'll never be cracked. I honestly hope that's right because I'm a little bit afraid of what happens if it gets fully cracked. That's probably a great place to leave it today. I appreciate a very collegial, interesting, constructive conversation. And I look forward to formalizing one or more$1 bets. All right. I'm going to think about it. I'll send you an email if I have a good idea of what we're going to do. I think these things are great. I would love to do a couple of$1 bets. So to be continued, but for now, Martin Casado, general partner at A16Z, thank you for being part of the Cognitive Revolution.

1:43:23Thanks so much. This was great. Turpentine VC is a podcast from Turpentine, the network behind Moment of Zen and Econ 102. If you liked the episode, please leave a review in the Apple Store or rate us on Spotify.