The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI)

27:49and I think there's certainly a lot of great knowledge work to be done in the US. I think there's also, if you look at what the actual sectors of the economy are, so a large portion of it is healthcare, a large portion of it is education. Tech is certainly also a large portion that's kind of leading the stock market and driving the stock market. There's a lot of great people who are trying to use the new models to try to do things like develop new drugs or understand how to make a real impact in healthcare. Or if we can get robotics off the ground and do things like start taking, helping with some of the physical labor, maybe not necessarily building houses, but the day-to-day household labor that could be large, untapped portions of the economy.

28:36Those pieces are really great. You can almost start to see the first pieces towards it. But the self-driving analogy is really interesting to me because early on, I'll say early on in my PhD, I was quite skeptical about self-driving. So let's call this 2018, 2019. It felt like self-driving was always a year away or two years away. Or if you ask the experts, they say, oh, five years away. And then last year I rode in a Waymo. And today I just actually got access to Waymo on the highway. So now conceivably I could potentially sell my car. And I live in the Bay Area in California. I won't because I personally like driving.

29:14But there's a lot of the progress is funny in this way where it's kind of it's not there. It's not there. It's not there. And then one day something, a switch flips and then you're suddenly like, oh, not only is this thing pretty good, it's actually a lot better than the service that I'd get in an Uber or a taxi or something like that. That's a really exciting thing. If we see that happen, if we see that switch flip for like, you know, household cleaning or putting away the dishes or things like that. I think it would really be really exciting. It would change a lot of folks' perspectives. I'm not a roboticist myself, but I'm really watching that space with a lot of excitement.

29:48Dan, as a quick tangent, do you think we're evolving towards a multi-hardware, multi-chip kind of world based on what you see? You know, obviously there's been Grok and NVIDIA, there's Cerebrus, there's like a bunch of sort of special ASICs companies coming up from your kind of like low level in the stack, Vintage Point, what do you see? Yeah, that's a great question. So it's something that I spend quite a bit of time thinking about, more so I'd say on the lab side than necessarily on the industry side. Although of course we're paying close attention kind of on both sides. I think it's at a really exciting time where the NVIDIA chips are really strong, really reliable.

30:32there's a lot of software support around them that has built around. We're starting to see the same things happen, for example, on AMD chips with some of the research there. So on the lab side, we put out recently a library called HipKittens led by my great friend Simran Arora. And she was really looking at, okay, how can we take, what are the right software abstractions to program on these AMD GPUs? And it turns out they're not exactly the same as the NVIDIA GPU. So even two GPUs that have relatively similar specs, certainly compared to Grok or Cerebrus or Samba Nova or one of these other chips, even though they're relatively similar, they actually have pretty different software abstractions you need to use.

31:13And I think more people are getting excited by that and investing time and energy into that. We saw the Grok acquisition from NVIDIA. A lot of people are excited about TPUs today. I think Cerebrus and OpenAI just announced their partnership. So I think certainly it's going to be a wave of things coming forward that you're going to see a lot more. I'm sure NVIDIA will still do great and still grow beyond their$5 trillion company or whatever it is at the time of recording. But I think you're going to see a lot more diversity, especially around, I think, inference of the model. So training and inference are actually quite different computations.

31:55And as a result, you might actually want quite different chips to do it. On the inference side, you might want, for example, your models to live locally, on your phone, on your laptop. You know, my phone, my iPhone, which is a few years old at this point, is already more powerful than some of the GPUs that I had when I was starting my PhD. That growth of that hardware power is really exciting to see. And then you mentioned a second ago in reference to self-driving cars, that moment where things flip, switches, is turned on. Has that happened with agents already? You talked about software singularity.

32:30Are we at that moment for agents? Yeah, I think that so personally in my life, I'd say that moment was last June ish. So June 2025 was the moment that it really flipped for me to give some context here. So what I do in my day job at Together AI is we write a lot of these GPU kernels. I don't know how popular, but in the general ML zeitgeist, GPU kernels are thought of as kind of like the final boss of the thing that you learn how to program. They're very hard. They're very highly parallel. You don't write them like you have to write in C++, which is this old language that the old systems people used decades ago or whatever.

33:07They're not in Python, etc. When you're trying to hire for people who can write kernels, it's very hard. It's a very challenging skill set. It's certainly the tip of the spear in terms of the programming strength. And last June, we had this really interesting realization where we realized that cloud code, cursor agent, these agentic coding assistants were actually very good at writing these kernels. So there was one week where I think I wrote like three or four different features that usually would have taken me a week each. And I wrote all of those in a single day. And I was like, oh, my God, this thing is making me five times more productive as a kernel expert.

33:45I got my team on it. Now my team has all these really complex systems that they've built where they can write a whole feature that I think would have taken months of a whole team's time before. And this is kind of that final boss of programming challenge that was really challenging. So from our perspective, for coding, for this really technically challenging GPU kernel programming, it kind of crossed the Rubicon for us already. And I think, you know, I gave this talk a few months ago at Slush about what we're calling the software singularity, where we realized, hey, in terms of software engineering, even for these really niche skills, it's certainly better than the average programmer.

34:30programmer, it's at a place where it can accelerate the really expert programmer. So it's right now, as of today's recording, it's at a place where if I just let it on its own, it might not generate the right thing for you. But if you give an expert programmer this set of tools, they can go 10 times faster than they were able to go before. And I think that that's a really exciting place to be. And on the topic of agents, Tim, you just wrote another great blog post called Use Agents So be left behind. And part of what you talk about is coding agents versus agents for the rest of tasks. Where are we in that arc where agents are transitioning from being excellent at code to useful for the rest of our lives?

35:15That blog post flows also as a reaction to what I see is there are large productivity gains if you use coding agents for all kinds of tasks. tasks and as a professor you don't code that much you can actually code more easily which probably other professors previously would not do so easy now but yeah also for non-coding tasks is super useful and when i look at um the productivity gains that i have some others smaller like two or three sometimes it's like 10 times faster i do tasks 10 times faster the quality is not degraded sometimes the quality is um higher um an agent might not be as good as i am but the agent gets get tired.

35:55Agent doesn't make bad mistakes. And you see, you cognitively struggle with complicated information that you put together similar to booter kernels, but what Dan mentioned, all of that is working. And I mean, Matt, as you put it, is coding agents and agents for other stuff. But how I would see it is, it's just coding agents. Coding agents are general agents. Coding agents can write programs that solve other problems. And code is so general, if there's a digital problem, you can solve it for code. And coding agents make the thing so easy that now you can solve a variety of problems in a way that you couldn't solve before.

36:37And this angle makes you productive. I would say this is the main thing that has been changing. Coding agents allow you to take a problem in a way that you could think before. and it's at a pace that you could think before, you can paralyze a lot of different tasks. The agent doesn't get tired. You just keep going. The work's much easier. One bit I love in your post is that you're careful to separate the hype from reality at the beginning, but then quickly you land from your experience of experimenting with agents, for the Lystra in particular, you land at the conclusion that more than 90 % of code and text should be written by agents.

37:15You need to do so or you will be left behind. I think for many people that are engineers, that's already true. And there is sort of this thinking, oh, if you produce text or code, and everything's done by agents, it must be low quality, much too bad. But the key thing is you inspect the code, you inspect the text. You might make some slide edits. The 10 % that you do might make a big, big difference. Through this basically sort of editing, just reviewing of output, you kind of make it your own. AI-generated things are not less personal than things you've written on your own. I see that. If I write a grant proposal with the help of an agent, it's sort of alive.

38:03I can feel like it's exciting. Like this person reading it said, like, this is good research. I want to fund this. I think that's just the reality. Like if you just generate things and don't look at it and just say like, yep, that's good. That will not help you. But you can quickly review content. You can skim it. You can look at like, ah, this doesn't look right or I want it different and edit it and you're good to go. That will be the reality. And the skills that you need to work in that way, they are not fully developed for most people. They're also not fully developed for me. It's still sort of a phase of experimentation.

38:41Models move, frameworks move. And so you need to adapt. You need to learn. There's a lot to learn. But if you do it, payoffs are huge. And I think there was the thinking that software engineers will no longer exist. But I think people no longer believe that. It's like software engineers are so productive. That is essentially what you need to learn. If you use agents well, you can do so many things. And I think that's the core thing. If you don't know how to use agents well, you will be left behind. That will become a critical skill. Practically, how do I do that if I'm not a coder and I think about automating some parts of my job?

39:21What are some of your recommendations on how to approach that problem? Yeah, I mean, the best thing is just sort of being sort of very pragmatic. Just think about things and trying to code them. Particularly if you're not a coder, that's very difficult. and there's sort of this barrier with, say, like, I haven't coded before. Like, I don't know this. But if you interact with agents, they can just build stuff. And with minimal learning, I mean, they can also explain stuff. With minimal learning, you can get there, execute programs, build websites. Particularly if it's visual, you get quick feedback.

39:55It's not that difficult anymore. I mean, often I mentioned you need to inspect things, but if you build simple tools for yourself to make your life easier, Often you don't need to do that. The agents write good code. If you work in a company, you need to integrate it in a good code place, you probably should review it. But if you build a small program on your own to make your work more productive, that's easy. Just a defining example that might be relevant here. I put, for example, a tool that if I have a video where I talk, so I record videos of how I interact with agents, then there are certain phases where I just look at outputs and try to understand things.

40:34and their faces where I talk. So I just build a tool that recognizes the speech that when I'm saying, it's a timestamp, then it slices the video. So basically you have an entire video where I talk rather than sort of moments where nothing happens. And that's very easy to do. I built this like in 20 minutes. I think everybody can do it because I didn't look at the code, the agent that did it. And then I looked at the video and like, oh yeah, it's still right. If you get started with a feedback loop, you don't need to code. you just need to inspect the output that you can understand or learn how to execute a Python program or a bash style and you're there.

41:10How do you pick what you want to automate? Like, how do you, how do I think about automation in my life? Yeah. So, so I also talked about that in a blog post and it can be sort of a sort of a more intuitive thing and then a more nuanced thing. I think the more intuitive thing is like, just think about what could be useful and then can That even be like something more complex. You say, I want an Android app or an iPhone app that does this thing. And you initially might think that's complex, but then you throw a coordination and it works immediately. The world's noisier. It's like there's so many things that you can just do.

41:48And you can be very creative and say like, what I always wanted to have and it wasn't there. Nobody built this product. Can I do it now? And I think that mindset gives you useful things. That makes you more productive. but it also flexes your muscle and sometimes it doesn't work and then you understand like okay at aging cycle this or this is what i still need to learn to make both these kind of things and i think that is sort of the more intuitive perspective that's very useful and that quickly get you started on the path where you say like um i mean first there's excitement then it's sort of sober there's a sober reality but then you pick up again and it's like you realize okay if I do it like this, I get more and more productive day by day.

42:30That's the more intuitive part. The more nuanced part is the part that I learned in the automation industry. I worked three years in automation industry in Germany, automating factories. It's a very calculated approach where you look at how you work, you time each of these steps, and then you say, if I automate this exact step in such a way, what could be the payoff? How much time would I save? And then you calculate what is the productivity gain? And then you calculate how much time do I need to develop this automation? And if you do that, you can quickly realize that automating certain things will not make a difference.

43:12The blog posts are matching emails. It doesn't really work. And there might be other things. A big thing is always calendar invites. Nobody likes to create invites for a meeting. But then if you think about it, you're also very particular about meetings. Some days you want more meetings, you have a meeting day, or you say, I can put this in before lunch. An agent doesn't know that. And if you specify that to an agent, you could also just create the calendar invites, just meeting pipeline. And it doesn't increase particularly much. And so there are a lot of problems if you think about this nuanced way and can say, okay, I don't do that.

43:49Ah, this will help me. And then on your end, what have you learned or observed in terms of agents? What works, what currently doesn't work, but will work soon? How to manage them? I think there's two broad things I've noticed for agents. So the first is making the agents effective ends up being a lot like managing junior folks on your team or at a company. So, for example, the new intern who shows up on your team, you're not going to go to the intern and say, hey, go fix our revenue for the year, double our revenue for the year or something like that. Maybe you'll try that once, but you're unlikely to see the payoff from that.

44:31Instead, what you often do with junior folks is you say, hey, here's a first little task that you can do to get to know this complicated code base. And here are the things that you might run into because you've kind of done it before. And when you give the agents that context, give them that ability to look at those things, then they can usually figure things out. The other bit is that when you have a new person on your team, you maybe won't give them access to all the production credentials and all the production database and all those things, but you're going to give them enough tools to be productive.

45:03So sometimes there's this tension between, oh, I don't want my agent to go delete my everything in production. So I'm just going to have it be hamstrung and watch every little thing it does. Whereas if you did that with a person, you would never expect that person to be productive. That's another bit. You kind of want to think about the agents as, at least today, as maybe interns or more junior folks. The other really interesting thing that I've noticed, and I think when I think more the educational role of a professor and how do you prepare people for this future where agents are going to be such an important part of workflows, is that how do you train for that?

45:39And one of the things that I've noticed is that the more expertise somebody has, so whether that's in, for example, Tim with process automation or expertise in what I do in kernels and writing these very highly specialized programs, the more expertise you have, the more powerful the agent makes you. And that's because you can work at such a higher level of abstraction. You know what the important things are. You know how to set the direction. You know what the common pitfalls are, like what is easy, what's kind of hard, what you need to break up into multiple steps. One piece of conversation that was coming up for a while was like, are agents going to replace all software engineers or things like that?

46:17Or are they going to replace all junior people or something like that? I think where we are today, that's probably, you know, clearly not the case anymore. where, you know, if I have a tool to make my team 10x stronger, I'm not going to fire nine people on my team. I'm going to say, okay, go do this, become 100x more productive than you were before. That's one bit. But then also kind of the script for how you become an expert at something is probably pretty similar to the way that it was before. You're going to study things deeply. You're going to try to understand things a lot. You're going to want to do things yourself with with your, you know, hands on and really get things done.

46:55In this world, you know, the chat GPT can teach you a lot of things. I was personally, personally, I was trying to get chat GPT to teach me all the little ins and outs of how a car works. I don't know how effective it was so far, but, you know, there's, it's, it's a lot easier to, to learn things now than it was certainly even five, two, three years ago. So those are kind of some of the, the, the things I'd say. So, you know, you want to treat the agents as if you're, you are in that manager role, you want to help them get unstuck. You don't want to just say like, throw the agent out of the problem, walk away and never look at it again.

47:29But you also want to kind of figure out how to level yourself up so that you can be a better manager, have more domain expertise and really understand things in a deeper way. So the fact that you need to learn and be an expert and that doesn't change, I think that's very interesting and that makes a lot of sense. This question is like if you show up on the job as a young kernel engineer and that's your first day, typically they would be, okay, well, you do this simple task and that other simple task. And then by year two, you graduate to a more complex task. How does this sort of hands-on job training look like?

48:04Right. Yeah. Yeah, so we think about this a lot together because we're still hiring aggressively even in this world where the models are very good, the agents are very good. The way we think about it, so first I actually went, the professor and me went and actually recorded a bunch of lectures on how GPUs work. So I make everybody watch those and then I still give them a task from scratch which is, okay, go take this flash attention kernel and modify it to do some other thing. You can pick the extra feature. The nice thing about the agents is that you can dive into that higher impact role in a way that you weren't able to before.

48:39So it's really impactful, I think, when a junior IC goes to try to manage someone for the first time, because you're suddenly starting to think in much more precise terms. So, you know, the classic software engineer thing is, hey, the PM asked me to do this and wrote this super long doc with all these requirements. But then the minute that you try to go ask someone else to do something, you realize the specificity that you need when you want to address a feature or something like that. The nice thing about agents is you can almost start to shortcut that process where you can have the junior IC still be an IC, still do IC style contributions, but they can now act as that manager role and act as their own PM.

49:18Because when you're communicating with the agents, you need to be as precise about what you're trying to get done. So in some sense, I've seen with the junior folks who are joining my team. So these are folks fresh out of college or fresh out of a master's when they are really gung-ho about understanding and being able to use the AI agents. They're able to communicate so much better than in the olden days. They're able to level up their level of understanding a lot faster. And then, of course, they can do things and build tools at a speed that would have been really, really hard to do, you know, five, ten years ago.

49:56And maybe I'll add sort of the educational perspective there. This I think is sort of quite interesting. It's a little bit sort of contrasting. So what is also quite interesting is the educational perspective on agents. I talk quite a bit that basically use agents will be left behind. And that is also true for students. But just like as Dan said, you need the domain expertise. You need to have some knowledge to use agents well. What we're seeing is if we allow students to use agents, they are very productive. but sometimes they build solutions that look correct that are actually very bad or just wrong and they don't realize we're at this point where it's almost very difficult to learn both domain expertise and agent use that's a very difficult balance to achieve because we don't want to have students that don't understand things but we also want to have students that and basically can use agents.

50:56And so if they can't do that, they will not be effective in the workforce. What Dan said is you already have a pretty good person with strong background knowledge, and then that person can level up their equipment with agents. But what do you do if you have someone that just learns computer science? How much AC should they learn? How much work should you do without agents? And that's a very tricky balance, and we don't know how to solve it. If we let people use agents, they perform very poorly on basic knowledge. And if we let people just do the basic knowledge, they don't know how to use agents and they can't compete.

51:32So they can't do useful work in the workforce nowadays. Maybe their solution is to all sort of the basic knowledge first and then agents, but that's not what students do. Students have access to these AI tools. They will use them because it's easy. And so maybe the solution is just you need a way of thinking of working with information and knowledge that you don't understand. And develop, I mean, there's critical thinking. I think this goes beyond critical thinking. You basically need to know the unknown unknowns, things that you didn't consider and don't understand and that you didn't even think about.

52:11You need to have the ability to think more about that to really keep up with agents. Because I think in the future it's realistic that we work on problems that we don't understand, that agents understand, but we need to keep up in some way, that would be difficult. All right. So to switch tags as we get closer to the end of this conversation, what are you guys currently working on? What's top of mind for you at Allen Institute on the one hand, together on the other hand, whoever wants to take it first? We have actually sort of a very exciting project that will be released very soon in the next weeks.

52:50And so I worked quite a bit of efficiency. I've been switching my work basically to coding agents. And so we will have a major release of an open source coding agent that has a couple of key features. For one, training is 100 times cheaper. You need to generate synthetic data and you need to train on it. And so we have a method that's roughly 100 times cheaper, but it still gets state-of-the-art performance. And then we have another sort of major result, which you can almost see as the holy grail of open source models. And that is, we can take a private code base. Like if a company, you have this code base.

53:33Cloud doesn't know your code base, but you have the data. You can fine-tune a model on it. So what we have is you can just point our method to that repository. You don't need to have any sort of tests or need to understand how to generate the data. It's just automatic. You quickly generate the data. And then you have an agent that is as good as a frontier model, but you can have like a 32 billion agent. You can depart locally. You can have an army of specialized models for particular tasks, for particular code bases, and so forth. And yeah, I think that is a very powerful result. All of that is also packaged with a science of coding agents.

54:15There are a lot of confounding factors, a lot of hidden things in papers that are not mentioned. We sort of unearth them, build scaling laws, and show what does matter and what doesn't matter. And if you put all of that together, I think the sheet agents, very few GPUs, everybody can use it. And very easily, we unblock people by revealing all the sort of secrets. I think there will be a vast change in terms of how quickly we can progress in coding agents. Very cool. What about you, Dan? Yeah, so together, I think the major question that we're trying to answer today is we have all these powerful AI models that can do all these amazing things.

54:53They're very expensive to run today. So, you know, the question in all the public markets is like, is OpenAI going to be able to turn a profit ever? and what's really exciting about together is that we are kind of on the forefront of getting these models to use the hardware as good as it can. We talked a little bit about training early on in the podcast. At inference time, when you have the model, when it's already been trained, already been post-trained, the hardware utilization is like less than 5%. So it's at a place where there's so much more that we can do. There's so much more efficiency that we can push out of these and we're really excited about figuring out how to use the hardware the best way that we can, whether it's serving customers like Cursor or whatever the next great foundation model company is.

55:37But yeah, we're really excited about pushing that frontier and then really getting us to this future that if these models are going to be as impactful as you think, if they're going to be running everywhere, running your daily lives, running your toaster, well, we better make it the best possible toaster that we can. Do you want to talk about megakernels and Together Atlas for a minute? Together megakernels, Together Atlas, these are both projects along these lines. So let me dive into the megakernels first. So to understand this, the first thing when we say kernels is we usually mean we are going to write a specialized GPU program for a single operation in a model.

56:15You can think of a model as one of these trained models. It's like ABC, different operations in a row, and there will be hundreds of these. And the way that we've been writing kernels for the whole history of, let's say, call it NVIDIA hardware, is that you really specialize a single kernel for a single operation. With these mega kernels, we're doing something quite interesting, which is we can take the entire model, however many billions of parameters, and put it into a single GPU kernel. And with that, you can start to do a lot more fine-grained optimization than you were able to do before. It actually starts to make the NVIDIA GPU look a little bit more like a Cerebris chip or look a little bit more like a Samba Nova chip in terms of the optimization that you're able to do.

56:55And this is really critical at inference time. So we're able to see 2x, sometimes 3x speedups over even highly optimized inference engines. So we're working on bringing that to really work in production, bring it to fruition and use it across our whole stack. Together Atlas is another really great, interesting research project that we've done recently where we can get the model. There's this technique called speculative decoding that we use at inference, where basically we have a little model that is trying to guess what the big model is going to do. And because of the ways that we've designed language models, if the little model guesses correctly, you basically get those tokens for free.

57:34So if you do the speculative decoding right, you can get, again, 2x, 3x speedups over just running a vanilla model. With Together Atlas, we do one extra thing, which is we say, okay, we can get that little model and we can actually adapt it to your traffic. The longer you use this model, the more it learns the patterns of what you're asking, of what you're saying, and it can actually get faster over time. So all these things, we have efficiency in mind, inference efficiency in particular, and we're all kind of pushing towards making all these things faster. What are you excited about for 2026 at a reasonably granular level?

58:14What do you think happens? What do you think doesn't happen? Yeah, so I think they're both sort of split. I think a lot of things will be very boring and not much innovation. But then we're also surprised by a couple of things that we maybe don't see. And I think actually sort of in the frontier, we will be less surprised. I mean, it's no secret that we ran out of pre-training data. And as Dan said, these are sort of the muscles. Then you can sort of smooth over. And you smooth over with synthetic data. And that's how you build coding agents on lots and lots of different environments. You combine the data.

58:52we make some problems there but I think you already see a bit of machine returns I don't think coding agents will be that much better the user experience will probably improve but you see it that all these models get almost equally good I had my config like GLM 4.7 set up and I used it and I thought I was using Opus 4.5 and then realized oh wait I used a different model because they're quite quite similar. And so I think we see less progress there. Where I think we see more progress is actually the small models. If you train smaller models on more specialized data, they can do quite well. And the smaller models that you get, they're pretty powerful.

59:39A hundred billion parameter model, you can fit it pretty well, even sort of low-grade data center GPU, like an RTX 6 ,000 or$6 ,000. I think for a lot of companies, it will be very interesting. They don't need to rely on the frontier models. The small models might even be better because they're specialized. The big problem, and Anthropic CEO pointed it out, we have these powerful open-rate models, but nobody uses them because the deployment is so complex. And that is because once you go beyond eight GPUs, you first need the users to make efficient, but then also very complicated inference systems.

1:00:13There's no open source system that can do that at the moment. We have disaggregated entrance, separation across sequence lengths, and so forth. Perhaps we can build this. We can build this also for an 8GPU machine for smaller models. And then the efficiency that you see with 100 billion models will rival what frontier models have. So you will get the efficiency of small models. You get the flexibility of small models. Performance on the frontier will stagnate, but on the smaller level, we get more and powerful models still, because you can distill from these large models into these small models.

1:00:47And taken together, I think that will change things. I'm also really excited about small models. I think we're going to see a lot more capability out of them. I'll be watching the open source models pretty closely, I think. With things like JLM 4.7, you're starting to see it. You're starting to see the open source models rival some of at least our current best frontier models. I think we're going to see another big jump in open source capabilities this year. I'm really excited to see new hardware. So we're starting to hear a little bit about Rubin, the next generation of NVIDIA GPUs. I think we're starting to hear a bit about the AMD 400 series of GPUs.

1:01:24I'm really excited to see kind of what that next jump in hardware capabilities is, even as we haven't fully kind of used even the current generation of hardware. I'm excited to see what people do kind of with all the other modalities. I think last year, video generation models had a little bit of a moment with Sora 2, with Gemini, and Vio, I think they called it. Really excited to see what they can do. And yeah, really excited to just see, you know, what is that frontier of intelligence that you can get on your laptop or on your phone and how fast can you push it, how far can you push it. I think it's never been a more exciting time to work in AI.

1:02:02You both mentioned state space architectures earlier in the conversation. Do you think that's part of the near future that we sort of evolved to post-transformer architectures with a state-space, JEPA, world models, whatever direction? Is that something that you see in the near-term horizon and that you think is desirable? I think in a lot of places, they're already there. So some of the best audio models in the world are at least partially based on state-space models. I think NVIDIA released a bunch of really great hybrid models recently. I think NemoTron is what they called them. And so there's a lot of really great work kind of there already.

1:02:43I think we will see the architectures continue to evolve. So in some sense, the DeepSeq MLA compression takes some of those ideas. One of the Minimax models had sort of a linear attention idea. So I think you're going to see a lot more diversity in architectures. You're kind of already seeing it. But certainly, I think out of the Chinese labs where there isn't really an open AI of China, right? So there's an open AI or a Thropic or a Google Gemini that kind of brings all these centers of product and model and revenue and all those together. So I think you see a lot more risk taking out of the Chinese labs where you're trying to differentiate the next model, your next open source model.

1:03:25One way to do that is architecture. Another way to do that, of course, is just pure quality. So I think we're going to see a lot more explosion of different architectures. All right. Well, this has been a fascinating conversation. I really appreciate the time and insight and thoughts. Thank you so much. Really appreciate it. Yeah, thank you so much, Matt. Thanks so much for having us. Great to see you, Tim. This was a lot of fun. Hi, it's Matt Turk again. Thanks for listening to this episode of the MAD podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from.

1:04:01This really helps us build a podcast and get great guests. Thanks and see you on the next episode.