How Foundation Models Evolved: A PhD Journey Through AI's Breakthrough Era

24:13with these types of programs you're building. So DSPy says, basically, there is a number of ideas that you need and you need them together, which is the thing that I think is a little trickier to a lot of people. You need these five things. There's five bets that DSPy makes and you need them together and you need to be seamlessly composable. And actually, in order to get all five, you don't need five concepts. You basically fundamentally need one concept. So the idea is we have Python and we have programming languages. These programming languages encode a lot of things that are highly not trivial.

24:41First of all, they have control flow. And control flow means that I can get modular pieces really easily because I can define separate functions. These separate functions and modules, the nice thing about them is that they really give you a bunch of stuff. So they create like a notion of separation of concerns where contracts of different functions can be described without you knowing everything inside the function or caring about everything inside the function. If you trust that it was built properly, you could just invoke it and it does its job. And then you can reason about sort of how you can compose these things.

25:11But you can also compose algorithms over functions. Like I can have a, you know, a more general sort of, you know, processor or function or something that takes these functions and applies things on top of them that are sort of higher level concerns. I can refer to variables and objects and mutate them or, you know, pass them around. When I say if this, then that, and I really mean it, I don't have to go to the model to reassure it that if it doesn't do this or if it, you know, if it does listen to that if statement, because I really mean it, I will tip it a thousand dollars. You know, and one thing here is this is a really limiting paradigm.

25:42Conventional programming is a really limiting paradigm. Why would we want to go back to it? And I think the answer is like all of the things I mentioned now, like all these symbolic benefits from a specification standpoint, now this is not about capabilities, are really hard to encode in natural language. You can reinvent them. You can tell the model, you know, if you see this, then do that. And the model might reasonably say, well, you know, he didn't actually really mean like 100 % of the time. I think the reasonable thing this time is an exception, right? Well, you actually can't do that with natural languages without implicitly creating a formal language.

26:16I mean, the most obvious version of this is ambiguity, right? So the dog brought me the ball and I kicked it. That's fundamentally ambiguous. You don't know if I kicked the dong or if I kicked the ball. And both are totally reasonable depending on the person, right? And so at some level, English actually doesn't do the job. Right. So, but programming languages, I agree, right? But programming languages are also really fundamentally limited in that you have to over-specify what you want. 100%. You kind of have to go above and beyond what you actually want because no ambiguity is allowed. And that forces you to think through things, you know, maybe you don't even know how to do it.

26:53Like, how do you write a function that generates good search queries or that, you know, plays a game well? Or, you know, it's very difficult to do something. Yeah, I don't want to get too wonky because I know where you're going. And I just have to say this because it just helps frame this conversation. I mean, by the way, what you said on X, which we're getting to really kind of changed my brain, which is so for imperative programming, that's absolutely the case. Right. Which is you need to know everything that possibly happens. Right. Or if you don't know, you make it you're you're making like the language is going to make a very fixed assumption.

27:24It's going to make some basic assumption. Right. So it's almost like if you're managing a state machine, you've got to know every state machine transaction. That's imperative. Declarative languages are quite different. Right. So in declarative languages, you actually specify what you want formally, right? And then the system kind of figures out how to get to that end state, right? But the problem is you have to be able to specify every aspect of that end state perfectly, which again, for some problems, is very complex. So that's also limited, and you just have to know the end state. So now you're working on DSPY, and I would love...

28:00You know, you talk about how using LLMs with a bit more formalism pushes it to yet another level. Right. So the only new abstraction in DSPy, and it's incredibly simple, it's just this notion of signatures. It's just borrowed from the word for function signatures. Our most fundamental idea is that, which is just so basic and simple, is that interactions with language models in order to build AI software should decompose the role of ambiguity. You should isolate ambiguity into functions. And what do you want to specify a function? How do you declaratively specify a function? I think the first, the most fundamental thing is it takes a bunch of objects.

28:40They better be typed and they better have interesting and meaningful names. It does a transformation to them and you get back some structured object, potentially carrying multiple pieces. And when you do this, it's your job, and this is not easy, But it is your job to describe exactly what you want without thinking particularly about the specific model or compositions you're thinking of. And this is actually a lot harder than it sounds to most people. So, for example, you would not, you know, so there is a class of problems for which some people actually write prompts that are almost signatures.

29:17So these are cases where you only have one input. Your output is just a response. You're not trying to like, you know, like you basically take a chat bot, you know, because the APIs are usually or the models are structured such that this is a very natural use case. And people like they try to prompt minimally. Right. So they don't encode a lot of, you know, they don't say, I don't know, think step by step or you're an agent that's supposed to do this or they just kind of just say what they want. So there's a class of people that almost implicitly write signatures. But there's something wrong with the original with the fundamental like shape of the API that usually exists.

29:45And so signatures are just saying it is a better shape. And we made every decision here slightly more carefully. Now, once you have signatures, every other part of DSPY from an abstraction standpoint falls off of it. There's really nothing else. Once you have signatures, you could ask, I have a function declaration. It's just declaring a function. It doesn't do anything. One of the hardest things about like people wrapping their head about DSPY and signatures is a signature does absolutely nothing. And it's entirely their job to build it. We actually can't help them at all build the signatures.

30:11A lot of the time people are like, well, couldn't you generate the signature from this or that? The signature is encoding your intent. I know nothing about your intent up front. That's the whole point. Wait, what are the, so what, just to be very clear, What are the signatures written in? I mean, fundamentally, it could be a drag and drop thing. It could be a, but usually it could be whatever. But the point is, it is a Python class usually. It's formal. It's not English, right? Well, it's a formal structure in which almost every piece is a fuzzy English-based description. So you could say something like, I want a signature that takes a list of documents and a list of documents is a typed object.

30:45You could actually say list of document and you have to define what the typed document means. And the fact that, the fact that this type is document, maybe the name matters. A list of documents is not necessarily the same as a list of images, right? They're different things. And they're semantically and fuzzily different. And basically, it says in English, given these inputs, you have several of them. I want to get these outputs and you have several of them and maybe the order matters. So it's really just like, I argue, it's what a prompt is supposed to be or what a prompt wants to be when it grows up.

31:13It really is just a cleaner prompt. Now, if you grant me that, which I argue is like a really small, it's a very simple contribution. There is really not a lot of richness to this, but that's the point. You get everything else that makes programming great while being able to build really powerful AI systems because you can now isolate your ambiguity at the right joints. You have a notion of like where you want the joints to be and the rest of your systems, the rest of your programs can be very modular. You can have multiple signatures. So now you get what people call multi-agent systems. Multi-agent systems are just AI programs in which you have multiple functions.

31:42You know, it's not really that. It's really nothing. It's not really a complicated idea once you take this. You get things like inference time strategies. People are like, chain of thought, you have to write your prompt in this way or we have to train the model in a certain way or React agents or program of thought. We recently released this thing called recursive language models. The thing is when you're solving a task, none of these inference strategies should be of your concern. It's if when you wanted to, this is just a thing that should be compositional and signatures have the shapes that we're like, we can use programmatic sort of constructs to compose over these types of constructs.

32:16When you think of DSPy, when you're originally creating it, and now as you think about it, do you think about it as something that will fundamentally only be consumed by humans or for humans? Not at all, no. I can imagine cases where you bridge the gap. Yeah. So the reason I ask is there's this obvious question is if the interface LLMs is going to be all automated anyways, do we need to enforce these restrictions that are primarily to keep natural language speakers within certain boundaries as opposed to if it's, you know, like whatever, and it's an agent calling it, we may not need to do that.

32:53So I think it's just, I think the argument in DSPy is intent should be expressed in its most natural form. So that's the declarative part. And the second part is, unfortunately, or fortunately, in the general case, that cannot be reduced below three forms. Some things are really best expressed as code. And no amount of automation can remove that. There's no amount of automation that can remove the fact that I actually want to think about three pieces because they're separate to me and I want to maintain them separately. No amount of automation is going to remove the natural language piece. Nobody wants to write Python to describe a really complicated AI system from scratch.

33:28And no amount of automation is going to remove the fact that for some classes of problems, you really need a more RL-like standpoint where you have a distribution of initial states or inputs, and you have a way of judging them or metadata about what correctness looks like, because that really captures the wonky and exceptional long tail set of problems that actually vary by implementation or by model. Yeah, you may also just want diversity. Give me something that may solve this problem, right? Like it may just be that like, there is no formal specification. Yeah, totally. Right, right. So there is a machine learning, like people associate DSPY a lot with the one that is most different to what they usually see, which is optimization.

34:09So a lot of new users and a lot of people that look at the paradigm and try to critique it conceptually, they miss the fact that you have to have these three pieces or like, In the general case, you can't get away without any. Now, by the way, there's a lot of applications where you do not need all three. If you're building yet another RAG app and the model has been post-trained to death to take a context and answer a question about it, you don't really need a lot of, you know, a lot of that to express your intent because just close to what the model is good at. Anyway, a lot of people associate DSPY with the third one, which is the database optimization.

34:38And actually a lot of well-intending users would write overly simplistic and general programs and try to distill their intent through data or through kind of this process of trial and error. And that's a really like bad sort of, it's like a misuse of the power of the models and the power of the paradigm. Because if you know what you want, nothing can express it better than just you saying what you want. The database of optimization is there to smoothen the rough edges. It's for you not to have to maintain laundry lists of exceptions. I'll wrap this up quickly, Martin. The other part of DSPY is the reason we built all of these abstractions and we haven't been changing them.

35:13This has been the case for, these abstractions are basically three years old. They've basically not been changing. And what we spend a lot of our research time on is building algorithms. And the thing about those algorithms is I'm not wedded to any of them. I rarely go out. I mean, we usually get excited about one for a month or something, but I rarely go out and get particularly excited about getting anyone to pick one of them over the other. We recently released an amazing genetic optimizer for prompts called JEPA. Before that, we had another one called Simba that was this reflective method.

35:43And we had Nipro before that. We have a lot of these algorithms. And they're really clever and cool. But the thing that I'm interested in is we build these algorithms to expire. As models get better, we can actually come up with better algorithms that fit turning the abstractions into higher quality systems. And what we want to happen over time is that our algorithms expire, we build better ones, but the abstractions that we promised and the systems that people express in those abstractions remain as unchanged as possible. So that's kind of like a, that's something that's kind of unusual to a lot of folks in the space.

36:19It may also help just to kind of pencil out where this sits in the software development lifecycle, right? There's two places you could put it. You could just be like, I am writing my software. I want to know what's the best prompt to use, you know, and then you could use it there. Or you could be like, actually, the best prompt is determined at runtime. And so maybe you could invoke this, you know, actually. So do you have, is there a standard use? Do you do this? like basically before this offer is deployed or are there actual runtime uses where you're trying to find the right? So the two sort of like concepts that exist in DSPy for this, and I don't know how technical we want this to be, but like we have the notion of modules.

37:01This is borrowed straight directly from like neural network layers or PyTorch modules, which is just saying, once I have the shape of the input and the shape of the output, which is a signature, I can actually build a learnable piece that has some inherent structure like what a machine learning person would call an inductive bias. And I wanted to take that shape and implement it for me, but carry some parameters internally about what it could learn. So that's a module. And a module is entirely an inference time object in the sense that it modifies behavior when it's being involved. So things like agents and different types of agents and code base or tool base agents or chain of thought reasoning, like all of these are inference time strategies that are modules.

37:40And the aspect in DSPY here is that these must be decoupled from your signature. Your signature should know nothing about the inference time techniques that you're using. The other aspect of DSPY is optimizers, which are, again, they're just functions, like modules are just functions, but they're functions that take your whole program, like an actual complete piece of software that has potentially many pieces. And they think holistically, how do I use language models in order to get this thing to perform its intended goal, which might be maximizing a score on a test set, but in principle, it could just be like, do what the model understands from the instructions it should be doing.

38:15And this could be, people do this at inference time sometimes in the sense that it happens while the user waits, so to speak, user of a system. But it's fundamentally a different contract because it sees the whole, there's extra information. I see the whole system when I'm an optimizer. I don't just see an isolated module. I can see all of the pieces. I can see a data distribution. I can see the notion of reward. And so I have much more, like a much richer space because there's strictly more information that no inference technique, no LLM sort of is able to capture just from an information like flow standpoint.

38:50You know, it's interesting because I mean, a lot of people think of DSPy as basically prompt optimizer, which is, you know, here's my whatever, my prompt template, tell me what prompt would be the best. But to hear you describe it, it's almost, you know, it's like this kind of, you know, declarative, you know, runtime-y type thing. do you know what the standard I don't even know if you have visibility of this stuff because it's an open source project but do you know what the standard use is is it the naive use case which is largely prompt optimization or are people actually using this in more sophisticated ways I think one reason where okay so I'm very I'm very loud about the abstractions I talk about them all the time I give talks I think about them you scalded me on X about this oh sorry no it was fantastic it was great no you really corrected me listen I was one of the few people that really thought about it as a prompt optimizer I really thought, listen, I'm going to write my prompt.

39:41I'm going to do some templating magic. I'm going to give it to DSPi. And then it's going to give me like, what's the best thing for what I want to accomplish. And then I'm just going to stick that in my program. That's the way that I thought about it until you made the point that it's actually more of a set of abstractions that will evolve with your program. So I tried to learn from what happened historically in computer science. Like you had these machines and you got general purpose chips and people were programming those directly in whatever language they spoke, right? machine code and maybe you could abstract it slightly with assembly.

40:11But then there was this amazing time where a lot of languages culminating maybe most popularly in C, but various others before it, got this idea that there's actually a general purpose programming model. You could build a model of a computer without thinking about any specific computer. And actually, that's a bit of an illusion because every specific computer is a lot more complicated. But you could create this illusion that is much more portable and is much closer to how humans think. And I know it's funny to use C as close to how humans think, but it really was a fundamental jump. Once you have C, it's important to ask why do people use C instead of writing assembly?

40:42And it's really weird to me that anyone would use C because it's faster than assembly, the code runs faster. So to me, when someone says they use DSPy because the quality of the system is higher, which by the way, is very often the case, it's not really the right answer because you're jumping, in my opinion, to a higher level abstraction such that actually I would be willing to give up some speed in order to have the portability, the maintainability, the closeness to how I think about the system and I want to manage its pieces. Like there is a trade if I'm willing to accept. Now, the reason people actually have universalized C and they don't regret it is this amazing compiler ecosystem where people build all of these optimization algorithms and passes and sort of infrastructure.

41:26You know, you inline functions. So you break the modularity, right? People are writing modular code, but you're actually breaking a lot of that modularity when it's being turned into an executable artifact, you eliminate that code. You have all these heuristics, different heuristics for different machines sometimes. And so like my vision here is if AI software is a thing and AI engineering is a thing that like needs to exist irrespective of model capability because we want to have these diverse space of systems, what is the abstraction to capture it? And if natural language is too ambiguous to be the only like the complete specification of these systems and it's too mushy and we kind of want to have more structure, well, what would that structure look like?

42:03And if we know what that structure looks like, well, if we do it naively, you would actually lose a lot of quality. If we build DSPy poorly, we might have a really elegant program that sucks, right? Like when you use it in NLM, it sucks. So the reason I build optimizers or like we build optimizers as a team is not so much that I think people can't write prompts and I want to write better prompts for them. What a boring reason. I don't care about that. Like people can write prompts. People can iterate on prompts. That's not an issue. The thing I'm trying to say is I want them to express their intent that is less model-specific and not worry that they're losing or leaving a lot on the table.

42:38Yeah. And honestly, this is where you change the way that I think about this whole thing. And so I'm going to try something. I alluded previously in our talk, but I want to try it again on this because this is kind of how it changed by thinking. You can tell me where I'm right or I'm wrong, which is... So you said assembly and C, but we've actually had a lot of paradigm shifts since then. So C, let's just say this is kind of like an imperative language where for every event that happens, you have to know how to handle it. Right? So traditionally in distributed systems, imperatives have not been a good approach because you could have some event show up at the node and then you don't know what state the node is in.

43:17I mean, the state space is so huge. And so you had actually a big abstraction shift with declarative languages. For declarative languages to be like, Like, okay, listen, we're going to tell you what the end state of the system is. And then the system will figure out all of the state transitions to get there, right? And so it's a higher level of abstraction for people not to have to worry about everything that kind of comes in and every event. And you can actually declare, this is, I think, like data log or something. You're like, here are all of the conditions that exist. And then I just want to make sure that the system is always in that state.

43:49And then the thing you kind of give up in that case is like, you can't bound the amount of computation needed. Like you don't know how long it's going to take to get there, but it'll always get you to like that state. So it's easier for a programmer. So like you can actually now build programs easier for certain classes of programs. Now when I look at DSPY, I feel like it's the same type of leap between like imperative and declarative, but for LLMs where there's certain declarative, like you can't write a declarative program that's going to solve the same problems that an LLM can because there's no fuzzy this and that and you can't really integrate them.

44:23Right. And so you want the same type of shifts that you went because you've got a new problem domain that you have. And so DSPY kind of gives you that with LLM. So you can kind of formally specify it in a way that's kind of natural, but also safe. And then it decouples you from the actual implementation below it. So is that a fair way to think about it? Or is that just a Martinism? No, I think that's a fair way to think about it. And I think one funny thing is, and I think you would probably agree with this, I don't know that declarative is better or imperative is better per se. Because of the problem domain.

44:53Like literally declarative is better for ones where like you've got a very complex system with a lot of asynchronous events because you don't need to maintain a state machine. Yeah. You know, all of these things have tradeoffs. All of them do, right? Like it was an LLM to like add two numbers. Right. And so I think like a really good shape for this is you want an imperative shell. Yes, by actually compared to there's a lot of sort of folks that create graph abstractions or whatever, like things are very declarative. I was happy to go on the record saying graph abstractions generally are a bad idea, in my opinion, for basically anything in computer science.

45:26But go ahead. Right. Exactly. And I think it's because humans, when they think top down, we actually think imperatively. And so like DSPy is just Python, which is, you know, I mean, it's a complicated language, but it's fairly imperative in that you're just like, you do this, you do this, do this. But at the leaves, where you were going to potentially have a fuzzy task, what were you going to do? I think you were going to write a prompt. I think the issue with prompts is actually, fundamentally, they're actually so declarative. They're too declarative that you're forced to break the contract of declarativeness because you're like, well, if I just say what I want, the model is never going to be able to fit in my bigger program.

46:01You know, one reason, by the way, people forget this is if you work with a chatbot that is tuned for human responses, you're doing most of the work that DSPy has to do in a program. In a program, if I have a function and I want to give it inputs and I want to get back outputs, those have to actually go into variables. Those actually have to go like, the output has to, so to speak, parse in a certain way. And I have to funnel things through this. If you're a human who's just asking the model questions, no matter what forming gives you, you're smart enough to be able to bridge the fuzziness in how the shape of the model.

46:38it's almost like the imperative is i know every step to the solution so do every step right and declarative is i actually don't know all of the steps but i know the solution so give me the solution and like dspi is almost i kind of know how to frame the solution do the rest of the work right it's like this kind of fuzzy or right right and i just listen there's trade-offs to each right i mean like yeah dspi like you have to like whatever have the overhead of a soda model which whatever took a billion dollars to train and it's expensive and infinite so these are just different points in the declarative space, the performance space, the cost space, etc.

47:10By the way, so I'm totally bought in. I actually think this is such a nice way, even independent of the SPI, like just the core work of this is how you should think about interacting with LLMs formally. I really think that you've kind of nailed that abstraction. So let's just take that as a given. So what are the hard problems now, or what are the next set of problems now to make that more pragmatic, like the optimizations under the covers or like whatever you need to do to kind of... So yeah, so everything we talked about today, I do almost nothing about this because this is work we did three years ago.

47:44And I'm just out there telling people about it. But I'm not really, you know, we're not changing these abstractions. What we actually do is the following set of questions. We ask, all right, someone wrote a program and we assume they did a reasonable job describing what they want. And maybe that means they wrote, you know, the control flow, they have the signatures and they have some data. These are the three pieces that you might want to have, or they have some, not all of these. How do we actually do a good job at optimizing this? So, you know, we've been, it's actually a really, I think, interesting progression to sort of see how we progress from the very early optimizers in 2022 to the latest ones.

48:15Very early ones had to work with models that basically didn't work, right? And we're not, had essentially no instruction following capability, but we're hit and miss for their tasks. So what we did look, you know, the reinforcement learning people do on LLMs, which is you take the program and you do what we call like we bootstrap examples, which just is another way of saying you just sample, like you just run the program, maybe with high temperature or something. A lot of times you see which things actually work and you keep traces of all of these over time. And then those traces, you know, which are generated by the model, they can become few shot examples.

48:48And if you just do that, sometimes it improves a lot. Sometimes it becomes a lot worse. So you just do some kind of discrete session top to find which ones actually improve on average. That was like when models are really bad. As models have been getting better, we've been moving, you know, we've moved a lot, basically all the way to reflective prompt optimization methods where like you actually go to the model and you're like, here is my program. Here are all the pieces. Here is what this language means. Here are the initial instructions I came up with from like just the declarative signatures.

49:13By the way, here are some rollouts that are generated from this program. Here are how well they perform. Let's debug this. Let's kind of iterate on the system to debug this. And obviously there's a lot of scaffolding to make sure that search is actually like a formal algorithm that is going to lead to improvement. But increasingly more and more of it is actually carried off by the models. One thing we also do a lot of is we ask, all right, conventional like policy gradient enforcement learning methods like GRPO. Nothing about them cannot be applied to a DSPY program because the DSPY program says nothing about how the optimization should happen.

49:45So actually, for a very long time, from February of 2023, you could actually run offline RL. And since May of 2025, you can run online RL of GRPO on any DSPY program that you write. People think that it's limited to prompt optimization, but I think the only notion that is fundamental in DSPY to prompt is that natural language is an irreducible part of your program. But that prompt is human-facing. It's how you say what you want. how it turns, how it gets turned into the artifact may well use reinforcement learning with gradients or natural language sort of learning. So we spend a lot of time on optimization.

50:19We also spend a lot of time on inference techniques. Like you just declared that you want your signature, which processes lists of books. Well, guess what? No model has long enough context to work with lists of books. So last week, my PhD student, Alex, and I released this idea called recursive language models, which sort of takes any model that is good enough and sort of sort of figures out a structure in which it can handle or scale to essentially unbounded lengths of context. And we were able to push it to 10 million tokens and see essentially no degradation. And the reason we build these types of algorithms is we really want to back your signatures by whatever it takes to sort of bridge the gap between whatever the current capability limit of the model is and the intent you specified.

51:04And the last thing we think a lot about is, well, we've made this argument conceptually and sort of tried to demonstrate it empirically that you need this irreducible, you need these three pieces, you know, signatures in natural language, structured, control flow, and data to fully specify your intent, at least ergonomically enough. The question though is, this is a very large space of programming where like you need to figure out how do, okay, I have a concrete problem. How do I map it into these pieces knowing that maybe I need all of them? And so we spend a lot of time, this is why it's a big open source project.

51:36It's like, we want to see what people actually build. and learn from that sort of what are the software engineering, like what are the AI software engineering practices that we should encourage and support. So these are the types of questions we think about. And I think one reason this has to have the structure of this like open source project, it's just like this large fuzzy space is, I don't want to be the only group or set a small number of teams working on any of these pieces. I think it's a space of the more academics and the more researchers and the more people work on optimizers, all programs benefit.

52:09The more people work on modules, all programs benefit. The more people build better models, especially programmable models, whatever that might mean in the future, it's sort of models that understand that they're going to get used in this structure, that everyone benefits. And, you know, it reminds me sort of the way in which deep learning sort of really took off, which was some people iterated on the architectures, some people iterated on the optimizers of, you know, you got things like Adam, other methods. And I think like that is what we're trying to really push a community towards. All right.

52:38So one last thing just to kind of finish off is getting a little bit more philosophical. But, you know, I think in AI, it allows us to do this. And what you're addressing is, again, the ability to declare intent, you know, for these models in a way that hits the abstraction, right? If you can guess prophetically, whether in the future, like the intent, like one of these models are going to have independent agency like agents or it's going to be humans guiding them. Do you have any opinion on like the direction this goes? I asked this question a bit earlier, but I kind of want to ask it a bit more directly.

53:14Do you think that the need for a human to declare things formally is going to go away? And over time, like we treat these like grad students or whatever. and this all just becomes the inner working of an agent? Or do you think that these things are formal software systems? This is a language like any other language and we will expect ESPY to be the interface. Something like it, something like that will need to be an interface that's exposed to humans for the foreseeable future. I think you need some amount of grounding into the world when you build these systems. I just think people in AI a lot talk about, we're talking about AGI, but this kind of just ethereal intelligence that just is so smart.

53:56But the problem is, you know, like intelligence that we care about, as far as I can tell, is really about, it's almost about the things you might want to ask or the way things are in the world. It's very world oriented. It's not really this, you know, it's not this very abstract thing. So as models get smarter and smarter, I imagine that a lot of the problems people write programs for today could get a lot simpler because that use case is kind of like risk versus assist architectures. As CPUs, if you believe in sort of complex instruction sets, it's possible that you had to do all of these, jump through all these hoops to do a fast square root before, but somebody just gives you an instruction for that.

54:40Models can keep absorbing with keywords or in their language more use cases, but the The human, you know, philosophically, as you say, like the human condition is that we will just want more complex things. And once you want these complex things, you know, in a repeatable way, you've got to build a system. And if you want to build a system, I don't really see that not having a structure. Like, I don't see that having the structure of LLM APIs today. I see it maybe some nicer facade on top of the... Maybe I can ask you a point of questions. You've grad students, right? Yeah. Do you ever wish that you had a DSPy interface to them?

55:20Like, a limit, it's a very structured way to have asks, right? And then if not, wouldn't I argue that you wouldn't want that for an LLM in the limit either? So I... I mean, it's a glim question, but I actually mean it seriously. Is it like humans just aren't capable of doing this stuff? And so that's the reason that we don't have formalism when talking to them? Or is it just a totally different... I promise this is an actual answer to your question about grad students. students. But to me, so here's the answer. The answer is the question sounds to me, don't you have chairs at home? Don't you wish that they all looked like tables?

55:52I need both. I really want to have... There's a software system. There's a grad student. They're totally different things. And there's nothing saying that AI that operates as a chatbot, as an agent, as an employee-like agent is a problem. I think we need it. I love... That's wonderful. Well, that's what it's told. Sometimes you want to specify something to a machine that has an LLM. And for that, sometimes you just want to talk to something. Two different solutions to that. I love that. That's a great answer. It's a great way to end this. Thank you so much for your time. This has been fantastic.

56:23Thank you, Martin.

56:26Thanks for listening to the A16Z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com slash A16Z. We've got more great conversations coming your way. See you next time. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.