Waste Tokens, Save Time

0:00Welcome. You're listening to the Naval podcast, your authoritative source for new knowledge. We're trying something new today. I have three frontier founders with us. Three good looking guys, actually, and a fourth good looking guy, Naval. And let me just introduce everybody. Guillermo the G Rauch. He's building Vercel into an AI cloud for the world of agents and whatever comes after that. Good to be here. Blake Schall, he's building supersonic aircraft in his own factory and jet engines as well. Blake's company, Boom Supersonic. And then Max Hodak from Science, he's building a biohybrid brain interface that grows living neurons on silicon to restore sensory functions like sight, but then eventually to explore new parts of the brain and new senses.

0:49All three of these guys are not composing their products with off-the-shelf parts. they're building their own factories and you know we don't care as much about what they're building exactly as we do about what they're learning about how they're building what's the new knowledge they're generating what's their alpha what principles are they discovering that other founders can learn from what are they trying to figure out right now and also what are the cutting-edge or crazy ideas that they haven't even talked about yet and they're still forming in their brains Naval do you have any reactions to any of that before I jump into Guillermo?

1:23Yeah, let's just have fun. Yeah, you guys should just jump in. Yeah, so I can't remember my exact quote, by the way, but I've been really pilled with this idea of software factories and the job of the engineer being something that you just show up to work, used to ship the output directly, and everything inside the company was, you know, how good is person a at shipping output b and now what's happening is the way that i'm judging you as an engineer is like are you producing the factory that will produce multiplicative outputs b through z right um and that's a that's a pretty significant change because basically like we used to believe and it should be somewhat controversial that there's 10x engineers like now clearly there's 100x or a thousand engineers and the world hasn't fully adjusted to this i used to get flamed on Twitter for saying they're 10X engineers.

2:15It flies in the face of so much like equality philosophy that everyone's equal. But the reality is when you're operating in idea domains, when you're operating intellectual domains and virtual digital domains, it's not even 10X, it's 100X or 1000X. And it always has been. Satoshi, Notch, you know, the guy who invented JavaScript, the Brendan Ikes of the world, John Carmack. I mean, these are 1000X programmers. Not to even mention, if you choose the right thing to work on versus the wrong thing to work on, that's an infinity difference. And it could just be not necessarily a better programmer, just one who had a better judgment on what to work on in the first place.

2:48And now, obviously, it's less controversial because of AI leverage. What's controversial is the token leaderboards, right? People are still getting a little confused because now they think, well, I have a bunch of 100X engineers. Look at all these tokens that I'm paying for. I'm curious if you guys have seen the same. How do you measure ROI? It's like the old measuring lines of code. You know, token consumption lines of code feel like similarly not direct paradigms. I mean, my observation has been that Claude or ChatGPT is basically as good as you are in a domain. And so if you're a really capable developer, then these things are really powerful.

3:29And if you're a junior developer, then you'll kind of find it to be like more of a junior developer. Like on the one hand, these models are incredibly capable. On the other hand, the feedback that you give them sporadically seems to be incredibly important. And the little updates seem to totally determine the types of performance you get out of them. There's a new kind of support that I give, which is you come to me and like you didn't get good output out of the model. And I tell you what to prompt the model with. So like the idea of like the quality of the reprompting, which I think you're alluding to is extremely important.

4:00But I mean, and to be clear, I think that this will become less important over time. Like as the models get much, much smarter, then you'll be able to put in less and get more out. But at least at this stage, it really seems to kind of reflect back the judgment that the user brings in, in my experience. I've kind of resisted learning all the tricks and tips. Like, you know, there was, oh, use Ralph Wiggum, use OpenClaw, use Hermes, use this prompt engine, use this scaffolding, plug in this piece, you know, always use plan mode. I just ignored all of that. I just assumed the model is just going to get better faster than I would figure out how to use it.

4:34It would figure out how to use me faster than I would figure out how to use it. And so I've just been completely ham-fisted with them. And I get frustrated at them and just sort of, I found myself typing less and less information and doing less and less work as time goes on with the models, because I just assume I can brute force my way through it. And I'll throw Codex, Claude and Gemini at the same problem over and over and just waste tokens to save time. and I think no matter how expensive these models might seem they're still way cheaper than a human so I would say just waste tokens save time don't look at the tokens either as inputs or outputs just look at your time and look at the final output and even if they're writing low quality code which I know in many cases they are it's not necessarily production quality or scalable code when the time comes and I want to ship it to production I'll just throw more tokens at it I'll say okay now go through look at it rewrite it and they're just going to get better every generation so yeah I don't see where this necessarily stops as long as we have verifiable domains and solve problems they're going to resolve those problems not in the unsolved problems domain where maybe you're Terrence Tao you're the cutting edge of creativity that you need to be you know working very collaboratively and carefully and closely with the model but I'm not in that I'm not at that level in software engineering and then Girobo you're probably the most extreme software engineer in the team right like out of this set you're probably the one who most hardcore came not from a software background.

5:56How are you finding these models at the edge of their capability? Well, there's one thing that's happened recently that what you're saying resonates strongly with, which is it used to be that you would give a prompt to the model and it kind of does it like classic like next token prediction thing. And it like runs away with your idea. And models now have been doing this like intuitive planning mode without, to your point, not even having to plan. where it comes back to you and says, look, what you're asking me for, there's these three routes we can take. There's this set of trade-offs that we're going to go down.

6:32That's the moment where people do the whole thing on X. It's like, oh, now we have a PhD-level engineer model. That's very clear that the models at some point graduated. They used to be junior engineers. Now they're principal engineers because they come back to you with a set of trade-offs. Obviously, sometimes they bullshit, which is hilarious. It tells you, this one is going to take three weeks. And this many talks, it tries to make really bad predictions. But clearly it's now this, like I respect the models a lot more as a peer, like that I'm going back and forth intellectually with. But there are a lot of gaps still.

7:08So like if you're a really, really proficient engineer or architect, I think you're still extracting more juice. So the question sort of that Max was positing of like, if you're junior, do you get junior back? Well, clearly not, because a junior gets more advanced knowledge in code that they would have never been able to write by themselves. But doesn't an experienced architect get 10x, whereas a junior engineer gets 2x? That's what I'm kind of trying to figure out still. Yeah, but I mean, I think there's architectural decisions. So when you think about the development, I'm seeing this now with some of the junior software engineers on the team of like, what is the next step in their career progression?

7:44it's going from like writing implementation for a feature to picking technologies like choosing between postgres versus some other database or picking between zmq versus some other message queue or like some other queuing system and those i mean the models can suggest them but that's the thing where you'll see it and you'll be like no no i want to use this other thing that's the type of little feedback that i'm saying really matters and the types of output that you seem to get at this taste taste and judgment right taste and judgment that said you can ask them which one should I use and why? And they know everything.

8:14They'll give you really good trade-offs. That's the change that I was saying has happened recently where you would say, hey, go and put this super high cardinality telemetry data into Postgres. And it's like, no, no, no, no, bro, we don't put that kind of data into Postgres. You should consider ClickHouse or Athena or whatever. That's happened to me a lot, which is really impressive. But the thing I'm still kind of struggling with this. Clearly the human is still completing the model. At one point, is it the other way about? The human is the one getting the instructions back on like, go get me this API key because it's something that only you can do.

8:54Or get me this amount of capital for my next set of investments that I need to make. You just watch. Clearly we're still not there yet. That's a temporary aberration. Pretty soon, every good SaaS company or hosting provider will have a CLI and API interface that the models can be made directly. They don't even necessarily need an API. As long as it's tech space, Unix space, the agent can hack its own API. And then the money part, you insert crypto tokens, put in Bitcoin, put in whatever, and the model goes and just pays for whatever it needs. And I think there are people working on this. But the thing I am now thinking through is pure software dead.

9:35like it's pure software engineering, like an obsolete thing. It's like saying speaking English, right? The models now speak English. We had to learn code to communicate with the models. Now the models speak English and they speak fuzzy, sloppy English, like a human and they understand things. So where's the moat like for a founder hardware, it's a boon, you know, like now you had to build hardware. It was hard to build a software company alongside like Patrick Cullison says software is art and it's hard to hire artists. So now as a hardware founder, great. you can have really good software develop fairly quickly.

10:06If you're creating models, maybe that's the new software engineering, training models and tweaking models and post-training and fine-tuning models. But classic software engineering is that dead, is pure software investable, is pure software, something you can organize a company, a team around and try to get some leverage. Did you guys see the, there was an article and asked by Mitchell Hashimoto called The Block Economy or The Building Block Economy, something like that. His argument is that the most useful thing for agents to have now is really powerful reusable building blocks. Because to Max's example, you wouldn't expect your clanker to reinvent a Q infrastructure system every time he needs to send an email.

10:48He needs to bring in the right building block that's right size for the task that you're asking for. And he say, well, okay, for this one, it's ball MQ. I challenge the notion that I would want the agent to reinvent the entire universe and first principles in a way that's incompatible with the rest of society and civilization. Like it's almost like reinventing highways, laws, policies, et cetera, just for you. Even if there is a potential for extra optimization, extra juice that you can get out of it, there's still a sort of like cooperation at large scale value of saying we're both depending on Postgres 13.2.

11:25And so that's still really, really, really valuable. I would say the category of infrastructure software and building blocks that these agents are going to use, obviously in BIAS is this where we're building, seems extremely valuable. And I don't see the agent anytime soon. And by the way, you could even, another metaphor that I've been using is like, any that's already been created that the models can reuse is like a token cache. Because you don't want to churn through a trillion tokens to reproduce what's already existing. And so there's always a starting point that the model can fork off from.

11:55But it's going to change things quite profoundly. So these are like libraries and dependencies, but for models? Yes, for agents specifically. To Naval's question though, I mean, I learned to program when I was really little. And that was the thing that through all of being a teenager and in my 20s, I get sucked into it and just code for 20 hours. And it was super fun. And I knew all this stuff about programming languages. I haven't written a single line of code in quite a while now. and I mean partly that's because my job is different but also since December I've built a huge amount of software that I now use every day there's all these projects that I've kind of fantasized about for years that now I'm like using that I've actually built and I didn't write any of that and I just can't imagine going back to like actually writing code by hand anytime like I mean I'm unlikely to do that anyway but just like in general I see that I have a hard time seeing that as part of the future yeah there's something really cool is that you understand how the pieces click together like I feel like anyone that understands what an API is and how data flows, inputs and outputs, performance, because you have to orient the model around this is a certain level of expectation that I have out of this operation.

13:06That's always been infinitely more useful than writing code. I feel like a really good, proficient engineering leader has been, quote-unquote, vibe coding through people on Slack or one-on-ones because you're transmitting your will, your intent, your experience, and you're letting others run with it. It's just that now we do the same, but with agents. And so I think that's why you've been successful with it, but I don't know that everyone sees the same level of success. I mean, I went from not having written code in 20 years to I'm coding all the time now, but through agents and I'm building tons of software.

13:43And it turns out that just understanding the basic principles of software engineering and algorithms actually gets you a long ways. Because the reason I stopped coding was because I didn't have time to figure out the latest language, latest architecture, infrastructure pieces to plug into. And I know Vercel makes it a lot easier. But even then, just getting started was a bare, like just plugging pieces together, assembling infrastructure, was just so annoying. The thing that really changed is, I mean, it used to be that you could build a lot, like there's a lot that was straightforward, but then you would hit some random thing.

14:14And then you could spend kind of some indefinite period of time debugging some narrow thing. and now with the agents what happens is you just don't get stuck anymore which is pretty amazing or they get stuck it's removed well no i mean like relatively quickly they can find like the right way to do things and it used to be that like i remember when their friends learned a program be like nope it's just like intrinsically frustrating like if like that's part of the deal that's how you learn and that just isn't true anymore