Why Agents Are Stupid & What We Can Do About It with Dan Jeffries - #713

30:35I mean, you can ask Anthropik and OpenAge. So they just put out, you know, Claude put out called computer use. And the first time I used it, I was like, oh, man, it's so fast. It's great. And it just ripped through one of my test prompts. And then another test prompt where I asked it to go to Google Flights, first of all, I couldn't get to Google Flights. So it decided finally to just go. It didn't realize that the screen was too small. And so the European zone pop-up button was hidden below the screen. And so it kept trying to recycle that. And it was like, OK, I can't get there. I'm going to kayak.

31:08GPR will be the undoing all agents. It's the end of mankind. It's the end of agents. It's like the EU AI Act. I just saw the EU as like doing this giant supercomputer so that everybody who trains something on there can be immediately classified as a systemic threat so it can't be deployed in the EU. Anyway, the... So...

31:34I lost my train of thought on this one, but... I think anthropic. Yeah, so the anthropic thing. So then I put it through another prompt, and it's going through that. Then it had a pop-up for a Google login. They covered part of the numbers, and they didn't know what to do there. So it just completely failed, whereas my agent kind of ripped through it. And I realized, again, they're throwing a billion dollars in computing, trying to get it to learn this, and they're going to throw a ton of more at it. And so is Google, and so is OpenAI. Great, because these equal better capabilities for us, which is wonderful.

32:12But our sense, my realistic sense is that we don't actually get to 97 % accuracy across e-commerce site, right? That what you want to get to is 97%. Like when something that's kind of the old school reliability test for software, right? If it's at 97 or above, then you're willing to deal with that 3 % of errors. And like this is, you know, perplexity feels like it's 97%. I don't, it's not actually like sometimes it doesn't, but it doesn't just error out on me. Like it returns an answer, right? And, you know, maybe it hallucinates once in a while, but in general, it feels like it's closer, but most stuff is not, is not there.

32:50I am not sure how much compute we have to pour into it. And the other problem is a lot of the scaling stuff is moving towards test time compute, right, or like thinking longer. The problem is, of course, with an agent and you're clicking around on stuff, you don't want it thinking a long time. You might want it in the planning stage, but you want it to be like motor memory, like you want it to be able to do stuff like in the way that you do stuff, right? So, you know, our feeling is that actually for a large chunk of time into the indefinite future, you're going to need like task-specific adapters or whatever in order to be able to do a lot of tasks.

33:28And I think, you know, I think my friend David Ha was talking about this, right, with Sakana, where they started like pushing towards, you know, evolving smaller LLM clusters, you know, to do a bunch of stuff. And I, you know, it's, of course, preferable to have the super generalized model that does everything. And that is just auto magically awesome at like clicking on GUIs and running around in the factory. But, you know, it's the internet is such an open ended, sparse reward system. right that it's it's incredibly challenging i think we're probably going to be in the point where you need to you need to train it give it specific insights and and for for a while in fact i'm betting the company on it so i hope i might you know but but but but but i you know and and you want them to get really good as a baseline right um but i think there's also going to come a pushback where people are going to want some of these things to be open source too Right.

34:28Because, you know, you know, a closed source proprietary model is a window into your life when you're like having it use your computer and do tasks in a way that we've never had in the past. Right. And I think you're going to inevitably see laws passed and spying and that kind of stuff and just hacking, straight up hacking. And so I think a lot of people are really going to want not just individuals, but corporations are going to want kind of open source models that they can fine tune on the test. And by the way, at a corporate level, if you're using some sort of internal application that you developed that doesn't have any kind of history, you're going to have to show it how to use that application.

35:12It's like we spent$5 million building this internal application that's like this legacy app that's been with us forever, but it does what we need to do. We're not replacing it. It's not just going to automagically do that. And there's a delta where the big labs are telling us that the models are just going to get so good that you're just going to plug them in. They're going to automagically do everything. But the reality is that in business, there's always friction. There's always friction. You put it in there. There's legacy systems. There's permissions. It's what it's allowed to do. There's culture.

35:45There's data that's sensitive and not sensitive. There's a million things that I think that's where a lot of the rubber meets the road kind of in the coming years. And I don't see any magical solution to get around it at this point. Now that, uh, that either queues up or runs ahead of my next question, which was, um, you know, when I'm talking to folks about agents, I often comment that I think a lot of agent, a lot of the things that, uh, we want agents for our political problems and not technical problems. And an example that I gave is like, for years, you couldn't get a Southwest flight to come up on Google flights.

36:23That's not because Google couldn't get the data if they wanted it. It's because Southwest would sue them and has sued people. And, you know, there are lots of examples like that. If you wanted to build an agent to give you, you know, the lowest cost for a medical procedure in the US, good luck with that. There's no transparency there. Um, and so, you know, at the risk of asking you a question that's directly tied to your dinner thesis, like, do you see, you know, how do you see when I think of this, it's, you know, there's one road that's like the momentum behind AI and agents and all this stuff like breaks down walled gardens, you know, and then there's this other more, um, you know, somber theory that's like walls are really resilient and have proven themselves to be resilient over time.

37:16Like, how do you think about, how do you think about that in such a way that you can spend so much of your time assuming that, you know, you can break down these walls, I guess? We can break down these walls. And I would say it's, it's an eternal cycle of something there is that doesn't level wall that wants it down, right? That was Robert Frost, right? And, And then the wall gets built up again. If you look at the early internet, you could search and find anything that you wanted. And then all of a sudden, more and more rules went up, more and more things that you could or couldn't do. More and more sites started to get taken down, right?

37:51These types of things start to happen. And then we're starting to see certain things in the internet as well, right? Like people used to have all open free APIs and all of a sudden, the APIs are charging. And like, maybe you can't do what you want in the API because they decided you can. You can't tag someone on Twitter or on LinkedIn, or you can't use the LinkedIn one without begging them for the possibility, right? And they're not going to give it to you because they want to protect their business. But you could hire someone for X number dollars an hour to go on LinkedIn and click to their heart's content and get that information for you.

38:24So I think it's coming to this pendulum that swings back and forth where exactly what you said, where we want to take it back. Like, I don't, you know, once you go, and I don't know what economics replaces this. I know it'll figure it out. But once you, you know, decide what you want to make for dinner, and you can just ask the system, and it's like, here's how you make sausage and peppers. I'm never going to that ad-laden and shittification level, like, website, right? Where there's literally, like, 10 pop-up ads, and it's an ad every other paragraph. and you know there's a long intro of like when my mother used to make this and i remember back in the day and i'm like i just want the thing and there wasn't there was again that goes back to the old school part of the internet too right where like they thought so much about how do you make ads tasteful where do you put them so they're not bothering people and for a long time that's the experience of the net where you would go out there and you do those types like nowadays it's just like how many can i cram in here as much as possible i don't care what it looks like i don't care about the experience the content is relevant people want to they want to take that power back right and so that's i think where a lot of the our friends at open interpreter you know working on kind of a desktop control and and what they want when they when we ask perplexity i ask perplexity eight percent of the time because mostly it just gives me the answer that i want then i can go look at the individual sites right to see what i need but it's like if i have to search myself i'm like oh my god i feel like i'm going back in time it sounds like your experience with perplexity is much better than mine well i mean do you got pro version or do you got like you just i mean i was using pro until i stopped using pro because and it wasn't the generation it was the search like they're they're they would well it was both they would come up with bad sources and then like tell stories about what those sources that were saying that if you actually clicked through and looked like, yeah, that was not in that article.

40:24It was just plausibly in that article. I think it's certainly gotten better over the last three to four months, like being able to set the model and then they kind of train their own model to be good. So, you know, from my standpoint, it's been, it's maybe have to go back to it and check it out. Yeah. I give it another shot. I talk about that as a risk of the hype cycle is that you try something and it's like, yeah, this isn't really working for me. And then you like, you know, put it in a box and don't remember to come back and check it. And it evolves quickly as a lot does in this space. It evolves fast.

41:00Right. And so like UIZR is another example. So it's a UI building system. And the first time I looked at UIZR, I was like, what? UIZR? UIZR, like UI lizard. Not a great name. Great product. Right. Sorry guys. I like your product, but it's really, challenging because this happens every time i tell someone about it anyway um so but really you guys are doing great work so but um when i first looked at it i was like wow this is what is this it's terrible um but then i got in there the other time and i was able to like you know just prompt first of all it has perfect ui design tools or gives you perfect react components they look great it's almost like using figma and then you can pop up a prompt and say like okay put in the screenshot and they'll just recreate that thing.

41:45So like in the middle of my designer working on something, I was like, okay, I want to redesign all of these screens. And like, he was asleep, right? You know, it was like, I was, cause I was traveling and I'm like, I just imported his screenshot and it like remade the screen. And I'm like, just rearranging everything. And I made like 10 different versions of it. And I was able to like, give it back to him, you know and these are actual like usable components, right? And that's super cool. high fidelity yeah like actual use them actually now as opposed to wireframes yeah yeah exactly like a full interface yeah so like i use it now a lot of like a lot of time just for prototyping out a cool interface right and you know and if i you know i can show it like another interface as inspiration and it'll like be able to kind of clone that thing up which is super cool and then i can just start pulling the stuff out and arranging pieces of the way that i like so i think that kind of remix aspect of ai and where we're getting to it's look this is the same thing we experienced in in every one of these kind of diffusion of innovation curves or in this hype cycle where it crashes and comes back out and you come up against the friction of dust and like and pebbles and everything else right it's like there's all these wonderful capabilities that we have to drag out right and yet you have to you have to learn how to build the tools for them the tools still feel the still the tools still feel primitive i mean even when i'm like you know i'm using ms swift which is alibaba's you know training framework right and like you know it's it's got a ton of stuff on the back end and you know has multiple inference engines vlm and lm deploy and because you need six of them there's no unified one anyway and there's like unsloth on the back and PyTorch and a million other things that they wrapped.

43:35And so you have this command line list that's like 400 parameters. And so then it's like this black magic of trying to figure out which of these parameters have made my training run crash, right, or sticky ball, right? And then it's like training run bingo, right? You're like, get the slot machine, wait 10 minutes and see whether it works. I thought all that would have been sort of finished. Is that part of what you're doing proprietary and and the i'm asking because i'm really curious about all those pieces and what you're doing there and if it's not proprietary in the sense that you would share it and folks are interested in the sense that they reach out to me and say so like we can come back on and talk about that yeah i mean so so in the ms the alibaba training framework stuff yeah i mean that's me getting involved in in kind of training side of the house to understand what my engineers are doing with this stuff because it's like um so for instance on the um on the gui agents we've been fine-tuning momo which came out of the amazing allen institute and they're doing the very hard work of open sourcing the data set the training scripts and everything which which the industry's gotten so hostile to this right both from a from like a proprietary corporate sort of standpoint of like keep your secrets and also from the crazy crazy kind of doomer narrative of like it's good we're and so like people are just afraid to like open source anything or like get attacked from many angles so kudos to them for doing that work but mo mo for instance is about the 7b is about uh 83 accurate on clicking on guis and the the 70b 72b is about uh 99 accurate now uh there's still room for improvement across the board and so yeah we've been doing a bunch of fine tuning with Agent Tutor.

45:24So you can basically show it a task and then we have the sort of a Rust space back in that can kind of just kick off a fine tune. And so part of it is like, we were trying to figure out, we have our kind of training system that we build in Rust, but then it proxies a lot of other stuff, right? And so one of those things is MS Swift, some of it is VLLM and some of the other open source frameworks for inference and training because you don't want to reinvent the wheel on that, right? So, yeah, the idea is to kick off these kind of rapid fine tunes and then test the model again. So basically, you know whether the model can do this task now and got better at what you want it to do.

46:06And right now, we're trying to find the balance of how much data it needs to be good at both clicking and sort of workflow. It's outside of the kind of reasoning stuff currently. we've got some additional stuff for thinking there, but we're trying to attack tool brain and little brain, kind of tactical and the clicking stuff first, because the reasoning we're hoping that the frontier models basically act like a GPU upgrade or a Linux kernel upgrade. Like you just kind of swap it out. Right, yeah, the big brain part. That said, we are thinking a lot about, and we've done a bunch of experiments around test time compute and that type of stuff that we're talking about too.

46:51I think actually the open source community desperately needs to put together an awesome reasoning corpus a la O1, right? And because I actually think that's one of the places where the open source community can get ahead of the proprietary lab by building a large enough synthetic data set or a reasoning data set, reasoning corpus. I think it's the type of thing where as a community, the open source could really I could really iterate and kind of get ahead and build benchmarks around it that start to show some of the stuff that's being hidden away, like the way that it's thinking about stuff. And you're beginning to see that with questions and things like that, but it's still going to be early for that part.

47:32So we're not tackling that yet. You said something that suggested to me like imitation learning. At least that's how we might have approached it before. Or like, you know, take a video of a user interacting with a site and then like train some model based on that video. Are you doing that kind of thing? Yeah. So there's sort of imitation learning, but there's also like we've tried a bunch of different techniques, actually. One of the funds I found that was really fun is I built kind of a prototype interface in-house that was kind of like Loom. That was sort of a precursor to AgentUner where you could just show it how to do something.

48:10And then I found if you use the Gemini long context window, one to two million, you can just upload the video and give it a JSON format for what the memory structure should look like. Not bounding boxes, because our models are better at doing that than those models were. But give it back an actual timestamp set of actions and a description of those actions in the format of the memory. Oh, wow. Which is super cool. And then essentially from there, you can insert that as a synthetic memory into the database of the system. And then what we would have the agents do is basically call up a series of successes and failures.

48:54So you would also be able to annotate it. So you'd be able to annotate to say like, hey, this was a success, and you should pay close attention to that. and then the system in memories after it had finished the task when when the agent would try to do something we could label that as a success or failure and give it a bunch of feedback that would be stored with the memory so you had kind of these synthetic memories which were like the human kind of showing you what to do and like so like and then what we're getting to now is sort of two different things so with agent tuner it's sort of an extension of the idea where you're like you're able to like show it what to do and it keeps it captures the screenshots the click data and the kind of workflow, right?

49:36And so it puts that in as a synthetic memory or as a fine-tune job. And then the next phase to get to is sort of the video, right? Where you can basically take it and kind of drag it and say like, from this here to here is this action and I'm annotating that sort of sub-action or the kind of overall thinking and pay close attention to this, you know, that kind of thing. So yeah, there's a lot of, there's a lot of, quite frankly, it's still the bleeding edge of all this stuff so we're we're figuring out as we go and like half the time we're testing the hypothesis is we're building the thing right and that's that's sort of the fun of it um and uh and in kind of the the part where i think we are in the age of discovery again and it's you just have to be willing to work with a model that came out last week and try crazy stuff and think outside the box every every single day that's really where i'm living right now.

50:29Do you have a methodology or framework and really nothing that quite that strong, but like, I'm curious how you think about, you know, there's 50 models that you can try for any given thing. Do you, you know, try three to try to get a sense for if it's feasible at all, you know, and then try the 50 or not try the 50, get close enough. Like, how do you think about, um, you know, that the next level of detail of you know dealing with the the crazy times that we have you know where there's so many options they're all evolving quickly there's like you know leaderboard leapfrog that whole thing there's not so many options this is that's a fool's game right it feels like there are um but it's just a lot of noise like let's be honest there's maybe a hundred useful models in the world um but across proprietary and closed um when a useful new model comes out it is it is painfully obvious, I think, to anyone who is actually good in the machine learning community.

51:32And like our, our team and the other teams that we respect are living and breathing that stuff. And they, they read the stuff that comes out and they're like, that's cool. Right. That's useful. That's valuable. It sounds like you're saying in practice, you are not finding a lot of like local maxima or something like that, where a specific model, you know, is really good at, you know thing x but you know maybe not necessarily noteworthy on other things and you have to find you know for each model the thing that it's really good at and plug it into your system you're more taking the approach of you know finding the best general purpose thing you know maybe scope to some degree or another but like and going with those things based no based on the task i mean so as soon as the mobile model came out we were like this is awesome this is you know this is you know because we i i immediately tried it out and i i uploaded the screenshots that i had and i was like okay find me this button find me this button and it was it had already in their demo had kind of returned a red dot where the coordinates were and i was like this already looks really promising this is our next fine tune and we had we had worked earlier with pally gemma because pally gemma was sort of built for fine tuning and had already been trained on bounding boxes we built a data set by combining data sets cleaning them and adding our own sort of data to it called the Wave UI dataset.

52:53And we had managed to fine-tune Pally Gemma to be Forex better than it was at returning bounding boxes. So we got it up to like 63 plus percent. And we released that model. So yeah, we're always looking for models that have specific capabilities. Like right now, I'm on a desperate hunt for models that store memory with much higher resolution. So everybody uses Clip. Clip is 256 by 256. 256. You lose a lot of information when 256 by 256. So now you've got some of the newer stuff coming out that's at 512 or whatever, 486 by 486. I'm waiting for the day when it's just not compressing it down, right?

53:35And that's sort of all built into the memory and you have a pure multimodal memory. So I'm always looking for that kind of stuff. The whole team's looking for that kind of stuff. So yeah, we're very much looking for specific models that do very good things and then obviously when it comes to reasoning obviously you're just looking for the models that have you know the best capabilities for us you know any multimodal reasoning model is obviously the greatest you know is the is the golden goose right that would be the one that we'd love to have but but you've got to have lots of little models to do things and like finding the right the perfect multimodal rag model or finding the perfect click model finding the perfect one that could do planning on things yeah you have to look at all that stuff and you have to be paying attention to it and there's no there's no kind of perfect source for that other than reading archive and twitter and talking to people and like you know the discord communities your your machine learning people are in and that kind of thing um but the vast majority of stuff that comes out is noise it's not you know it's like well we're like three percent better than this or like we quantized it down to like you know it's like you gave it a lobotomy it's it's useless like yeah it runs on your desktop, but who cares?

54:46The fact is there's maybe a hundred models in the world that are really valuable. And then maybe there's a couple thousand proprietary ones for individual tasks that people have trained that we don't know about or something like that. But in general, it's a lot smaller than it seems if you can see through the noise, I think. You spent some time working in MLOps and kind of thinking about the infrastructure. or to what degree are you, you know, what does that look like for you? Are you at the size where you care about that? You know, how, you know, how much automation are you trying to build around, you know, various things?

55:26Like, it's not super important for us now, other than when, like, we, when I find what I have found, it's funny, it's like, as CEO, like, half of my job is just leading the company. but like i love to keep my hands you know in the cookie jar playing with stuff and so like i was always i always wanted to be a good coder but i was pretty much a terrible coder i was a great sysadmin and so like so like you know claude and o1 and all those are really giving me a great chance to code and be like a great prototyper and i think differently than coders right so like my my genius coders would be like that can't be done i'm like i hate that word and And I'll like prototype it because I don't have any priors and 50 % of the time I'll be right.

56:11And they're like, God damn. Right. And then they'll go implement it correctly. Right. So that's fun. And then I'm spending a lot of time on the system inside of the house currently, just sort of like I said, I was playing with the MS Swift, trying to really understand the state of the art and kind of distributed training. so we can, not because I need, you know, a 10 ,000 GPU cluster, but, you know, I want to make sure that we can fine-tune the 405s and the 70 VMLs and that we can tune them fast and that they're valuable to us and, like, how can I make that a stable? And almost everybody in the company is kind of a Swiss Army knife.

56:45They can do a lot of different things. I was very surprised. I pretty much felt like when we came out of the MLOps era that MLOps was solved. i very much do not feel that way now like i i i look at this and i'm like wow we the the mlf era got truncated and rug pulled by the gen ai era and all the money went over to there but it was not finished um and and the problem is a little different it's different enough it's different enough like i think the main thesis of the mlf era was that everyone was going to be doing advanced machine learning. Big companies would have thousands of data scientists training models from scratch.

57:28That is never coming to pass now. It's the frontier models and the open source models that deliver a huge chunk of the functionality. You might train a proprietary model for a small subset of tasks or something very specific that you need to do that's not covered by those things, or you're going to be doing a hell of a lot of fine tuning and culturing in data sets so the problem shifted it just became it just became very different the thesis was wrong the thesis was just wrong and so now we're at this point where it's like wow i really want the thing where it's like auto magically take my data set analyze it and be like i have clean ditched i have i have split it i have put it into the correct formula that you need based on my deep knowledge of Lama 3.3, you know, and...

58:19Sounds like, what, Paka, Paka? Paka, Paka. What was that company called again? Packager. You can't get that from like HPE or something? I cannot. So no, those systems don't really exist in a way that I think is valuable. When you look at the open source community is proliferated, so when you start to look at inference engines, right, you look... There's four different major ones, and they all have model-specific code around them to make it run faster. They've done optimizations is the word I was looking for. So it's like, yeah, you can use a general-purpose one that'll run any kind of model, but it's just going to run slow.

59:06So it all depends on what that project is interested in. And it was very interesting to see, for instance, VLM was primarily interested just in the tech space models and not really in multimodal. And the Chinese were super interested in multimodal. And so by the time Momo comes out and they were looking to support in VLM, there was a big project internally to rewrite all the multimodal stuff. and like it kind of you know i could see in the prs it was like okay well almost almost raised for it but we're going to finish the rewrite of all that stuff right so there's kind of there's all this stuff that's sort of happening and it still feels very much like black magic right it still feels like why did my you know fully shorted you know fsdp pie torch run just fail on me the last few days and have me tearing what air i have left out you know um and it's like who knows man like you know i'm watching youtube videos on this like this is not auto magical at this point it's not doing like perfect error recovery and then spitting out a clean error that says like hey you know what you should really think about ratcheting down this this and this and i think the problem is this it's just sort of like digging through stuff uh oh i think that's the era right and like working with Claude on it to be like, and feeding it the docs and reading the stuff ourselves and watching YouTube.

1:00:28That's not where we should be, but that's where we are still. And it's frustrating, to be honest with you. Open source. Yeah. You've mentioned open source on a couple of occasions and clearly excited about it as a consumer of it. Are you contributing to it or are you planning to contribute to it with what you're building? Yeah. So we've already, we have, we've already released like 10 different repos as open source. We've got a K8s. We've got a Kubernetes-style deployer for agents. We've got a virtual desktop that it can control like a VNC, so the agent can control it like you would control a desktop via VNC, so it can spin up a container and talk to that.

1:01:07There's a tool fuse, which basically mounts that into the agent so that it can talk to it. And we've got another 15 plus repos, many of them which will be open source. and i think we have an open core philosophy 50 plus in the works mate is that i mean yeah 50 50 plus in the works you know that are sort of still behind behind the closed doors still okay um you know not all of them see the light of day obviously uh um but uh many of them will i i think we very much believe in open source open core and um it's not that nothing will be proprietary but to me open source is an unmitigated good for the world.

1:01:46I've actually been really angry that I've been fighting the open source battle again in AI. I thought that was one in the Linux era and that you'd have to be an idiot not to see that open source is the most important software in the history of mankind. It's worth trillions and trillions of dollars in value to the economy, the entire cloud ecosystem, your phone, the router in your house, supercomputers, everything. and yet here we are it's under attack again right with the same old stupid things you know like you know the bad guys can use it the bad guys can use everything right i mean this is this is ridiculous like i have a kitchen knife and we don't take the kitchen knife off the market because somebody stabbed someone we let the other 99.99999 % of people cut vegetables with it so i've been really frustrated to fight that that that battle again because i thought i thought it was over but i think you've seen this sort of paranoid uh you know sort of a dimmer crowd kind of come in and try to attack it and you know we can't do this like it'll it'll go crazy it will lose control and i'm like once again i'm like describing magical properties and stuff to me it's it's a hugely uh a huge loss for society when you go back to kind of the you know the the proprietary pure software play.

1:03:05Proprietary software always has a, I'm not, I'm actually, I'm not an open source fanatic in the way that it's like everything must be open source. You know, I'm like proprietary software is super awesome. It's super useful. I want them both to coexist. And I think that open research makes things more secure. I think it makes them more understandable, interpretable over time. Like that's where the research is going to get done. It's not going to get done when you get a paper that's published. It's like, we're not going to tell you anything about the how we restrain what the architecture is you know how you know that's you know you it doesn't matter if you have 10 000 people working there that's different from you know 10 million people looking at it in the open source way that's the reason that all of the you know encryption algorithms and stuff are open right and not proprietary right it's you've got to you've got to put it out there so to me i'm i'm an open source diehard it's one of the few hills that i'll die on i think it's absolutely essential did we cover all of the highlights from the talk anything we missed we covered a lot of good ground how about we look we covered a lot of it i think it's really i think this is just an exciting era i think there's there are ways to make the the we talked a lot about how to kind of make the agents smarter i think it's going to be a dog fight to make them smarter over the course of time i think i think a lot of the cool stuff that's been done with um reinforcement learning is going to start to really bleed into it we're starting to envision you know could we just clone up like you know 100 000 websites and let let an agent go wild in there with you know so yeah like alpha web right just going in there and just kind of playing around with it until it figures out how to use that right could that be done i think so right and i think that kind of stuff could be you know super awesome and really kind of take it you know to the next level but i think everybody building these things today i think to me the timelines are not as like compressed as people think they are like and you know dario and i are going to disagree on this where you know he's like look i've seen that movie before the scaling laws will just work and we're just going to throw you know throw more stuff at it and like it's absolutely going to get smarter i have zero doubt that they're you know produce better and better models but a lot of them are plateauing with what we we can you can see it in the charts are plateauing right it's like it's so it's i i i don't i think you need some fundamentally new stuff um to kind of make them better and i'm excited for that but i think you got to get your hands dirty and i think there's there's nothing like applied at and this isn't even pure machine learning but it's like trying to get it to do real stuff in a reliable way is is where the rubber meets the road i think it's where the next crash comes but i also think it's where i think it's where the multi-billion dollar applications come from too and the agentic era takes back some power for us like you were talking about earlier until the pendulum swings the other day again, and that gets in shitified in 10 or 20 years, right?

1:05:59But it took a while, right? It took a while for the dark side to manifest. And I think the agentic applications are almost limitless. Even at the lowest level, you think of it as RPA++, but at a high level, being able to do something super open-ended, we don't have any comparable technology. It's the most general technology you can possibly have. And why would it not be as important as the printing press or the internet or, you know, the hammer and nails, right? It's just a, it's a fundamental, and now I'm starting into the hype cycle a little bit, but I just, but I don't think it's next week.

1:06:38Self-awareness is important. I don't think it's next week. I think it's a gradual process of hard work. It's a Sisyphean process. It's really hard, you know, and the further you roll the rock up that hill, the harder it gets. And I think, you know, and you're going to have some plateaus and some peaks and some troughs. But when these systems are effective at maintaining sort of human workflow or doing human workflow in an open-ended way, I don't know what you asked, what are the applications? What are not the applications? Right? That sort of becomes the aspect of it. So I think it's just exciting to be in this age of discovery again.

1:07:14And a lot of companies are going to get whacked again when the music stops and the chairs get pulled up. But the next trillion-dollar companies, the next world beaters are going to come out of this era again. And it's going to be something really awesome. Well, Dan, as always, it was wonderful catching up. Awesome. Thanks so much. Yeah, man. Hopefully not as long this time. but you've got a lot of people to talk to. I've got to make myself relevant enough that you're going to ask me to be back on here every six months or something.

1:07:55Awesome. Thanks for having me. I appreciate it. All right. Thanks so much.

1:08:16you