What's Left for Humans When AI Builds Everything?

24:52They were going down the Disney path, you know, going to maybe make it even more accessible for people. What's the strategy you think over there? And then buying a nascent podcast, lots of weird moves that would lead one to believe that they're doing tons of side quests and then shutting them down. What's your take, Jonathan? What is the discussion at drinks at night or at pickleball or whatever when people are in and around these companies and have offers from them? Firstly, Jason, I have a ton of respect for both Sam and Dario. And I think the key is coding and enterprise. I think that's the answer to a lot of this.

25:34And that's the reason for all this crazy growth, the focus on coding and enterprise. And coding is key. I can't relate it out really beautifully, like where the models get better at coding, they get better at out-of-domain tasks in reasoning, math, STEM. We don't fully understand why. And people believe that it's because there's something about coding that teaches you to think algorithmically, teaches you to think step by step. Coding is lower ambiguity compared to other natural language. So there's been some research on how there is transfer out of domain with coding. With nothing else, this is true.

26:10I think coding and math. So I'm glad Karina is working on automating math. Coding and math, I think, are the key. The second reason, Jason, is when you win in coding, it actually accelerates AI research itself. What's the number one thing AI researchers do? Write code, implement papers, execute new ideas. So if you win in coding, you will just improve at a faster rate. That's reason number two. Reason number three is a lot of things that don't feel like coding are actually coding. For example, when you ask a search engine a question like, Like, hey, what are the key trends in AI investing in 2026 relative to 2025?

26:47What that search engine might be doing behind the scenes, it'll come up with a plan. It'll execute a query to PitchBook. It'll execute a query to Crunchbase. Write some Python code to analyze the data. Use NumPy, Matplotlib to plot the results and share it back to you. That's the model writing code. So without you thinking about it. Yes. A CFO that's preparing a board material could be writing code to calculate stuff like LTV, CAC, magic number, all these fun things. So coding is really, really key. And I think both OpenAI and Anthropic are really focused now on coding and enterprise, which is wonderful.

27:22And in consumer, I feel like the feedback loops are fairly well established. There's TrackGPT, there's Gemini, there's a few others. And when you have that feedback loop, when you know what real humans ask your AI assistant, and you know when you did a good job, when you didn't do a good job, you know when to work with companies like Turing to generate data to improve in the areas that you're weak in. And in the same way that Google had a lead in search a while back relative to Yahoo and Bing, Google's improvement rate was just faster because they had more usage. In consumer, the flywheel is set.

27:55In enterprise, the flywheel is wide, wide, wide effing open. It is open. And it doesn't transfer as much. The person that wins in financial services may not be the person that wins in life sciences or the person that wins in healthcare. But you got to deploy, you got to let the models touch reality. And if we are getting to the end game in terms of the amount of data and the training, and then it becomes a game of how do we make the tokens, Karina, less costly, because that's what everybody's looking at right now. And there's a report, we'll pull it up here. We'll show it on the screen. There's some leaderboard contest inside meta for who can burn the most tokens, which is like giving a marketing agency, like who can buy the most billboards and ads and TV commercials?

28:42And that's their success without ever looking at what did those commercials and billboards actually do? There's a report that, and I don't know if this tweet is correct, but back of the envelope math, they're like, oh my god is meta 5 billion 10 billion of this 30 billion dollar number and what is meta thinking here when you read this story karina is it incredibly stupid or is it brilliant because if your team embraces this and burns through a bunch of tokens well at least they're using the tools you got everybody off the bench to use the tools so maybe it's just like a crazy thing that you're just immersing everybody.

29:26Here's a tweet from John Chu from Coastal Adventures on Twitter. Plenty of my Meta friends told me folks have been building bots that just run a loop burning tokens as fast as they can due to this policy. It's an absolutely stupid policy and it's similar to how Meta uses LOC to measure engineering output. Managers are supposed to use it as a proxy and dig in to understand work complexity, but plenty of managers are lazy and just don't. Karina, your thoughts?

30:02Carina Hong:Yeah, that's fascinating. I mean, like I'm looking at this sort of like, you know, comparison marketing team who spent the most money. Oh man, I guess like buying like Michelin star restaurant, like the whole venue's table is what we'll do. But I don't know. I think like Meta, you know, this is a way to, I guess, like force like more AI usage and especially given, I think, like the sort of like push to really innovate and, you know, like use all the AI web coding tools. But like any metric can be gamed. I mean, like this is just like something where people have been like realizing since like the beginning of time is like anything that you say is like a metric.

30:41Carina Hong:And especially if it ties to a performance review PSC, then it's really survival at a game. And in a way, I think the big tech and especially the layoffs has been kind of in a squid game. I think things are rough there. We see the one direct indicator is the real estate market, like the housing prices of Mountain View is really going down. And it's so much worse than last year this time. I don't know. I think a lot of the sort of senior folks at Meta could use a little bit push of that. But also those ones, you know, beyond level five, they almost have guaranteed tenure in a way, even if they kind of like mess up, you know, one project or another.

31:20Carina Hong:It's more like the junior folks, sounds like those that like really have this hacker spirit and like build a loop. That's just interesting. Humans reward hack too. Yeah. Well, I mean, if you show me an incentive, I will show you an outcome. And if the incentive is I'm being judged by this, then all you have to do is, you know, if you want to save tokens, you say, hey, speak like a caveman. And B, just tell me if you got the task done, yes or no. Don't explain all the details. Don't show me your work. Just confirm it worked, yes or no. And then you could just say, hey, talk like an NBA bullshit artist.

31:51And it would just give you all this flowery language and burn up millions of more tokens. Somebody said the way they were getting clawed down 80 % was to just have it talk like a caveman this week, which I thought was pretty interesting in terms of the agent space, Kanjun. So explain what you think is going on here in terms of the embracing of these tools. And then maybe the difference between the levels of developers. Is this actually making, you know, a level, a junior developer move up the stack in terms of ranking? Or is this like AI slop? Like, what's the reality on the ground? Is it making bad developers like appear better, but they're not actually better right now?

32:37Like the game on the field in 2026. We actually went from having one product to 10 products in January. And we split our team to be basically max three people per project. And it's because coding agents are at a point, Cloud Code in particular, but also Codex actually, are at a point where you can write high quality code with them, like very high quality code. By default, they will mostly still write mostly slop. But with good infrastructure for running these agents, you can actually do a lot autonomously. So our team probably spends like, you know, about one FTE per team on token burn. Got it.

33:23So for every three developers, there's essentially a fourth in tokens. Exactly. Exactly. Does that seem normal to you or appropriate or does it seem crazy? I think it's actually low. So right now we're at a point where, so part of why we build open agent infrastructure is it accelerates us. Like you were saying, Jonathan, training models for coding accelerates researchers when they're trying to train more models. They write mostly code. Well, building agent infrastructure that makes you much better at coding and makes these coding agents run for a much longer period of time. We think general agents are mostly just going to be writing code a lot of the time to do what they're trying to do, because that's one of the most reliable ways to accomplish a task is to write code and then verify in code.

34:08And so a lot of what we do. Yeah.

34:13Carina Hong:Can you measure successful token counts? Like, you know, is there a way for them to refine this metric, like in your, in your expert view, you know, to, to measure like those effective tokens, whatever that means? Yeah, we actually, I mean, we're a small team, we're only 30 people. So we measure internally by like each team tries to figure out how long can we autonomously run these agents and have them produce PRs that don't need any review or editing. So we are looking at how many edits are made in a pull request. A pull request is when you submit code. And one of our teams is running, one of our teams, my CTO is running these agents overnight and in the morning waking up to about 60 or 70 pull requests and proving them.

34:58A lot of them don't need any changes and just pushing them. He's gone from, he's like writing almost 10 ,000 lines of code every day right now. I mean, lines of code are not a good metric, but I mean, that's just dramatic. That's like. And do you have a fear that we're now, because of this abundance, going to make incredibly overcomplex pieces of software, as opposed to elegant pieces of software? Because constraint made people write better code, right? You have only a certain number of developers and you have to be thoughtful. Hey, these 12 features, we're not going to do them right now. We're going to focus on these three features.

35:35What I'm starting to see is people can build so fast. They're building monstrosities in terms of products. So how do you think about managing that? Yeah, it's kind of like saying like, oh, because we now have a lot of consumer products in the world and we buy things from Amazon, do we build overly complex houses or interior design as opposed to like really simple bare bones interior design? The answer is like, yes, but it's like much more expressive, you know, as we've run experiments like refactor the entire thing automatically in 24 hours after writing tests, like full test coverage on the thing.

36:07That's something you can do now. So you can make your product really complex. You can have scope creep and add a bunch of features, and then you can just redo the entire thing based on what you learn pretty quickly. And so I think what we actually see is not junior engineers, senior engineers, junior engineers are being helped, but the senior engineer knowledge of how you design a system is still super, super critical. I think those things, the improvements kind of even out, but overall the workflow for writing code is changing dramatically. And I think this time next year, you'll see that the workflow for software engineering is super, super different and you do something else.

36:44How will it be different? And then Jonathan, I'll go to you to get your input on this. But Ken June, how is it going to be different? If you had to make a prediction, what in the process is the key change next year? One thing we're playing with is can we grow code? So can we have models be the ones like growing and maintaining code? We're playing with something where you can like text the agent and the agent will, you know, be writing the code and evaluating it fully and writing tests and then coming back to you. And so like people call it like, I think it's more interesting than just being a maintainer of a project, but like, can the model be the one to grow the project and experiment with things and try things?

37:24I think that's something that we're going to see more and more of over the next few years that like a lot of code is actually generated by the model. I'm curious, actually, Jonathan and Karina, how you use coding agents and like, how does that affect your engineering workflows? Like, I love how at the bleeding edge you are with, let's call it AI-assisted engineering. Like, what is your advice for are engineering managers doing product building today in the post-cloud code agentic era? Maybe top three takeaways from the way you run engineering. You said something about you running thousands of cloud code instances.

38:00I feel like you and the cloud code team might be at the bleeding edge. And Karina, I would love to hear how you run engineering at your company. I'm happy to share some stuff on my side as well. Yeah, this is not a product pitch, but we actually just open source our infrastructure for running these agents. It's called Manager, MNGR. We open sourced it last week. And it's a library for orchestrating agents. It's like a very simple library. So it lets you say, like, encode things like, for every test in the last week that had a flake, like, you know, sometimes it passes, sometimes it doesn't pass, fix that test.

38:36or for every user flow in my workflow, test that user flow every time X happens. So you can write these agent programs where agent running is part of the program. And the reason we open source it is because we really want other people to be building on this more open infrastructure. You can swap out the models underneath. You can swap out whatever you want. You can add your own memory, whatever it is, like store contacts locally, you get to keep your own data. Enterprises keep their own data. And so this lets you kind of programmatically string together agents and also programmatically verify.

39:15So we'll run this with stop hooks that say, don't stop until you've solved the entire problem. Don't stop until our verifier, we have another open source product called VET. It's a verifier for verifying coding agent conversation history. So it's like, and also other bugs and code that they produce in the pull request. So it's like, okay, if that finds any issues, like fix the issues before you submit the PR. So you can do a lot of this like if else kind of things programmatically. And that's what makes it really powerful. You don't want to be running the agent manually. You want to actually programmatically build up the system for you.

39:49And I actually think engineers and we, in the best case, we have exoskeletons where like we slowly built up this exoskeleton over time of like how to do the kinds of cool things that we want to do. like the knowledge work we want to do, the coding we want to do. And we can build that up step by step, block by block programmatically. And that's what we're trying to help people do. That's what we do internally, at least. I feel like the process of developing software has just changed dramatically in like the last two or three months, right? I feel like you should write a book. You should write like a book or a paper on like, what's the...

40:22We're hearing it across the entire portfolio. And we're seeing it in products that people are pitching us, which are, hey, I think it comes back to the recursive stuff that Carpathie was sharing, where he's like, okay, well, what if it tries an experiment? Well, in the consumer software space, you might want to try a feature, and you might pull that feature from the feature request that people are commenting. Or you go to a subreddit, and it says, man, I wish this product would have a way to split fares for Uber or DoorDash or group ordering. and it's like okay i read on reddit and in our customer support logs that people want to split bills okay let me research that let me write the code let me put it in there yeah again here's carpathy's uh you know running llms uh and over a couple of hours or a couple of days it looks like maybe two days there and making it better um yeah that scientific experimentation and humans are just really bad at keeping it up for a long time.

41:25They just, when you're running a startup, you just eventually are like, oh God, I'm so exhausted with these customers. I'm so exhausted with this product. And people can't stay focused. Yeah, we're not meant to be factors of production in an economy. We're meant to be living creatures, not like a productive item, but we've been made into these weird productive items. Yeah. And I mean, the entire economy, at least for our lifetimes has been how many people out of, you know, can we get 5 % of the people to do STEM, to write this code, and can we keep them from having any other life but submitting this code and making these products for the other 95 % of people?

42:05Now it's like that whole career is going to just be abstracted into a cloud of agents. Yeah.

42:12Carina Hong:It's fascinating. I think like from the sort of like hiring perspective, I had this interesting realization where, so we hire like a team of mathematicians, you know, we open up a London office, a bunch of people are programming and lean, you know, in the Imperial College London community. And so we hire them as like math experts, like there's sort of our in-house, you know, data labeler. And they also like look at like, you know, quality of output. But then we realized at one point, they all start like vibe coding and hacking our like action prover. So they kind of like, you know, just like shifted from like data labelers, experts, all the way to developers without us noticing.

42:48Carina Hong:And I'm like, what have you been doing? I'm like, I'm hacking, you know, Turkey and Turkey is the internal code name. So it's just fascinating. I think a lot of teams are kind of like, you know, interviewed and built in the way of how software engineering used to work. I mean, you have lead code interview. Now, I don't I don't know. I think like I heard places like Cognition, getting rid of those sort of lead code interview and like switch that to like either a work trial or over like long consecutive hours like show me something you can build with all the tools available to you and that's just an interesting change um i think like people interestingly i think talent market wise there are two groups of people who are extremely important one is like the the most engineering of the engineering folks like those are like the infra guru and they are just like in hot pursuit And there's this other like, you know, research scientist type talents and maybe not coding machines.

43:40Carina Hong:But now they because of the algorithmic mathematical way of thinking. And I think Eric Schmidt recently made the point is that people who, you know, grew up doing math are going to be more and more in hot pursuit in the software engineering market. Because you are going to have all the tools available to you, almost like automatically five research engineers to join your team where you are the lead scientist. So I think it's interesting how these two groups of talent, one very low level, one very high level. Jonathan, the velocity of this commodification is stunning. Because if we were sitting here a year ago, we were talking about making developers, are they going to be 5 % or 15 % more efficient?

44:21Because it's auto-completing the next sentence.

44:24Carina Hong:Right, right, right. And oh, how much faster will they advance in their career? And now we're kind of moving to, well, everybody's obviously a developer. and what's, two questions for you, Jonathan. How are you architecting it? And then I want to go around the horn of what's the end state here? What happens in two or three years if these things are recursive and able to run themselves? Like, what are we all doing in the world when it comes to software development? Like, what is the human role in all of this? I feel like the folks who are like starting companies today are so much more advanced with like how they do this.

45:01It's a paradigm shift, yeah. It's a paradigm shift. It's a literal paradigm shift, yeah. And I'll share what I'm doing differently at Turing. So firstly, like I've,

45:16I just feel like, so I'm trying to automate the job of the CEO at Turing, right? I'm sure you, both of you are probably doing the same thing. I'm certain of it. Very excited about this. And it's a lot of chores. being the same with a lot of chores. And I did this like over a weekend with Cloud Code. That's how magical this was. So I have this cadence where with my exec team, I used to have a chief of staff where I would have to think of what are the most important topics we should be discussing at the exec team level every week at Turing. And I would try to get what's red, yellow, green at Turing across each of our customers, especially what's yellow and red And what actions do I need to take?

46:00But I am fundamentally token constrained, right? Like I cannot read through hundreds of projects going on at Turing at any point of time, thousands of pull requests happening last week. I don't, what did we ship last week? Like we have a 100 % engineering team. There's a lot going on. I don't quite know what's going on. And I was getting information filtered through layers in the org chart, like managers telling stories, others telling stories, lots of synthesis. So I was like, F this, I'm going to go direct to the source. I'm going to look at actual dashboards, data in Salesforce, data in Jira, data in GitHub, all these raw inputs.

46:37I feel like the truth is in code and people talking to customers. That's where ground reality is. So I built the system, which automatically pulls in all of this and creates like a daily brief for me every week. What's red, yellow, green across all our customers? What are the topics we should be discussing as an exec team? And the topics are so good. And the exec team is finding it useful, right? They are getting more information. It's the virtual chief of staff. It's the virtual chief of staff who doesn't sleep and has complete access to everything. all information. That's the key is when you give OpenClaw, which I did, like root access to Google suite, I can say when I'm talking to somebody, CIR, and I made a little skill, check-in report.

47:26And I just say, when somebody's starting a conversation with them, I'm like, CIR five. And it just gives me a summary of their email by category, by priority, their meetings, and their start of day, end of day reports in Slack. And it's like, okay, I didn't have to waste 15 minutes, you know, just asking you and being a detective, what did you work on this week? Or what's your priorities? It's like, your priorities are what you did. So the whole game of interpreting and gaming each other, the employee trying to game the CEO, what does the CEO want to hear? Why are they asking me these questions?

48:02The CEO trying to pick apart without accusing the employee of just fucking off and not doing any work. You asked them, what did you do? People are like, why are you asking me? Am I getting fired? Or are you taking a project away from me? It's human nature. It's the Elon system. My underlying prompt is what did you get done last week?

48:25Carina Hong:But I'm new here. So are they okay giving you guys that access? I don't know. I feel like I'm wearing you. Small company. Yes, I can explain this to you, Karina. it's very simple. When you work at a finance company or you work in customer support, there is no expectation that what you're emailing, what you're DMing, what you're doing on your work computer, there's no expectation of privacy with that if you work at Goldman Sachs. They know every single thing you type is tracked because when they want to fire you, they find you saying some stupid stuff on the Bloomberg terminal or in the Slack and they record every phone call because they have to make sure that if something goes south, they understand like, oh, you shorted the stock?

49:07And it's like, yeah, here's the phone call. Here's the confirmation. You know, it's a highly regulated industry. The benefit of knowing everything going on on the computer, whether it's on Slack, whether it's on Zoom, is just so great for a manager. And then you just have to tell your company, you just tell your company, listen, this is going to advance the company. Don't do anything on your work computer, that's dating or your Coachella plans or whatever you're trying to do with peptides or memes. That doesn't belong on a work computer. You have your personal phone over here. You have your work phone.

49:42People are fine with it, I think. And if you're not fine with it, by the way, the paradigm just shifted. It's over. Every CEO is doing this right now. We had a CEO on the pod last week, he said, I created a script just to tell me every decision made in my company. So this was a very interesting angle of attack. Every decision that was made and summarize the debate, give me the decision made, and then rank them by importance, whatever. So now the CEO on Saturday can just say, okay, they made the decision to invest in this company, to not invest. They they decided to go Prada on this, pro-rata on this, not.

50:19It was Nick Harris from Light Matter who was on last week's episode, This Week in AI. He's working on photonics. It's incredible what he's working on. But he just said, I just need to know all the decisions in this fast-moving company, so tell them. It was almost like you pulling up pull requests or whatever it happens to be. So Karina, I give you permission to have God mode for everything. I mean, you have God mode for all the code written, right? yeah it's interesting everything is recorded now everything's in slack everything's recorded there's a middle ground there's a middle ground which is um for example in my system i call it enigma like the the the uh there are like 10 different meetings that happen that i actually should probably be on which are like account reviews and so on but if i did that i'll just be in meetings all the time right so we use granola so the meeting notes are transcribed that feeds into this, like in a meeting.

51:15And in a meeting, there's definitely no expectation of privacy, at least for the CEO and others. And specific Slack channels, like where the actual work happens, like humans talking to each other. So I wanted all the tokens of human interaction. So Granola gives me that. And then in each meeting, there's usually an artifact, like a Google Doc with like, what are the discussion topics? What was decided? So those I think are fair game. but as Jason said I mean it's in a company I think it's all good as long as you're open about what's being tracked just let them know people who want privacy in a corporate environment like this it's one of two things there's a valid reason like it's HR data and it's people's compensation or something but even then for the last 20 years we've tried to have an open compensation philosophy in Silicon Valley so like you don't have this weird stuff where or yeah yeah it's kind of trying to be more transparent makes everybody get focused on work and you know not focused on trying to game the system um and the second group of people are people who are around that's it like people who are messing around not doing their work a remote worker and i had this happen we literally installed a piece of software on the computers because we're a finance company that just tracked like to make sure people weren't exporting stuff on thumb drives because you could just export it on a thumb drive.

52:39In your case, it's the entire code base. In my case, every legal document we've ever signed, right? Or, you know, somebody's due diligence when we invested in their company. Like, this is particularly dangerous, right? I might have, like, very sensitive information about a board meeting where there's a lawsuit going on and that's not a public lawsuit or threats of a lawsuit. This is, like, really crazy stuff. And I just found out, like, some people were working an hour and a half a day. It was, like, really disappointing. Or the people who were like saying that they needed a raise, like somebody's like, I need to make$200 ,000 a year.

53:10I need to double my salary. And I was like, okay. And then they were like, hey, boss, she's working 30 hours a week. This other person's working 60. It doesn't make any sense. Like that would be completely profoundly unfair to double their comp. But other people are doubling their effort and they're more effective. So it's just you have to decide how do you want to run your organization? Do you want to have an elite organization like an NBA team? NBA teams. Kenju, and they will track people's blood work. They'll videotape them shooting shots and say, hey, here's how to increase your shooting percentage by 5 % if you do this protocol.

53:44And they're tracking their sleep data. It's just the nature of business today, I think. It's kind of over. Yeah. JC, do you envision as an investor, will you be looking at this type of data and companies you invest if you could do this type of, you could kind of, I mean, how many times they use the word align stakeholders like companies that use the word align. I mean, it depends on the relationship you have with your founder. So for me as an early stage, like high trust, you know, backer, you know, as opposed to like a late stage person coming in and wanting to mess with the business for some personal gain.

54:18I'm like, I'm always so early, 90 % go to zero. I make my money off of like, you know, an Uber or, you know, Robinhood every hundred investments. So I don't sweat the small stuff anymore, but yeah, for hand-wringing VCs who lose their minds and want to send CEOs and management teams on crazy adventures to do tons of reporting, like, oh my God, that makes you want to like shoot yourself in the head as a fellow board member who witnesses like that. I've had to take multiple junior, you know, associates on a board and just say, stop giving more work to the team. Let them cook. All that matters is what their customers are saying and the talent level here.

55:01Let's focus on those two things as opposed to you doing this crazy churn report and da-da-da-da-da. Yeah. I don't think I need access to that, but most founders, I just tell them, come to me with the hardest problem. So if you think about what this technology will enable you to do, people are taking entire board decks and board notes. This is investors I'm talking about. And they just say, prepare me for this board meeting. What are the questions I should be asking? So it's happening on the other side. So you all are preparing decks with AI and projections with AI reports, analysis. And then on the other side, the VCs are saying, OK, this is 200 pages worth of board materials.

55:42Tell me what's the most important things here. What should I be asking? And so eventually there will be a board member who's AI. You will put a C3PO on the board. You'll put a replicant on the board and they'll just be the greatest board member ever. Yeah. I think we should use Imbue. I think we should use Imbue and we should all let the AI agents have board meetings with each other. I think a shadow board is actually - You can use - We have a board meeting every day. You can use Imbue and have these agents - Agents can tell you how your company is doing all the time. Can't you? And it sounds crazy, but I think creating a shadow board and saying, this is the legendary VC who's made more money than they could ever spend.

56:23And they understand the essentialism of creating a great company. This is the tactical board member who understands go to market better than anybody. And, you know, this is your finance board member. They understand how to, you know, work with CFOs and sales directors. and you just have them grinding every day instead of like every quarter, every day on your startup and just being like in your ear. Like that's a genius idea. Actually, that's really interesting. I have a version of this, but I love this idea of making a board. I have a version of this that's like a group of advisors that, you know, are kind of, oh, this is like what this person would say, what this person would say in my advisor group.

57:02And I'll ask them every week, roughly, give a give a bunch of data on what I'm thinking about and get their perspective on each, you know, their AI mentor perspective. And it's really useful.

57:13Carina Hong:That's just fascinating. I heard there are like people on Chinese social media, they're saying that people distill their colleagues and people distill their like ex-girlfriend and like someone trying to like distill their colleague to get that colleague fired, which is like, it's like, oh, like, dude, I think you have an AI to like plan your episode. That's right. That's right. I was using this for white mirror. I would say that's the extreme of sort of this. That would be a little crazy. It sounds like a dystopia. So I had asked this question earlier, like, what's the end state here? I have some ideas around the end state as an investor and as CEOs and that whole dynamic of the startup game.

57:55I want to put that aside for a second and just say, what's the end state for product production and software development? Where do we see that if this continues in three years? Like two years ago, we're talking about making developers 15%, 20 % more effective each. Oh my God, we're going to get a free fifth developer if this keeps up, or we're going to get twice as many developers. Then we're like, well, each developer will have a 10-person team in tokens who are agents doing all the work for them, and they're just going to be doing air traffic controller or whatever. Okay, now let's fast forward 36 months.

58:29What are the possibilities here, Kanjun? If you were to just, if you succeed with your open source platform, you know, everybody's got 100 agents working for them, multiple simulations going, and we hit some level of super intelligence, what happens is the game of life just ends. every piece of software is instantly created and we don't even have to build startups anymore because people look at a glass mirror phone and say, uh, I want a meditation app suited just for me. And it just intuits what you want and makes you the thing. Well, where could this be in three years? It's a great question. Three years.

59:06And I know you were thinking about like the human freedom is like one of the topics we were talking on the group chat about. So you can kind of dovetail, well, I think that topic in here, which is purpose, you know. I can bring it in. I think there are, I think we're actually kind of at a fork in the road, and there are two default paths. And in the current default path where Anthropic is winning, OpenAI is winning, Google is winning, what we're seeing is AI getting integrated into existing products, their existing products, and we're seeing this verticalization, like Anthropic is trying to kill third-party, Anthropic cut off access to OpenClaw for, you know, maximum pro subscribers.

59:43And Sam bought the founder, I believe, to get him distracted from OpenClaw. Yes, that's right. Which is the worst thing ever. I mean, that was the most – I mean, in a list of sinister things that people accused Sam of doing, I felt this was like the most cynical thing to do. Take the most promising open source project and say, I'm going to give the founder a couple of hundred million dollars. I'm super happy for Peter. Yeah. In order to distract him. Yep. Yeah, but the way he's killing it. Like there's one thing you can buy a piece of – you can buy a company and then shoot it and like kill the product and kill the brand, you know?

1:00:22Like that's dark. Yeah, this is a very like sinister influence kind of behind the scenes. I think you're right about what's going on. But yeah, you see these – like OpenAI and Anthropic are killing their competitors because that's the capitalist system we live in. You know, they have to make investor returns. They have to make a moat. They have to lock you in. Right now, there's not much mode. They're losing money. And so it's hard. It's hard to operate in this incentive structure and not do that, not try to kill your threats. And I think the default outcome is that they keep verticalizing. You know, Max and Pro plans keep getting subsidized.

1:01:00Enterprises want to stay on their team plans or their enterprise plans because they're also subsidized relative to the API. and so as a result the like external third-party providers the open alternatives they are just more expensive for both enterprises and consumers to use and so you kind of end up locked in into this ecosystem they don't like to share their memories of you you know you have to like kind of claw your memories you can't download your entire conversation history from either Claude or OpenAI which is wild and so they know more and more about you and it's less and less easy to get out and as you said Jason they are going to understand you really well they are going to intuit your preferences they already do that, you already feel it I feel it at least I mean Karina is punching up her poetry with it that's right, Karina yours is really good at poetry your unpublished poetry I don't know what's in your unpublished poetry now I'm thinking those poetry are nerdy poetry

1:01:56Carina Hong:you know like I'm now like suspect that maybe Kod is not better at writing maybe it's just math poetry that Kareem yeah I know I was like oh wow what does that mean prove this theorem and make it rhyme

1:02:13no I think Kareem is a real artist a true artist but in the default path what's going to happen is that these companies we're going to give our digital lives and our digital identities to these companies and they're going to rent them back to us. These are like our digital selves being locked up and rented back to us in the default path. And we're going to like rent all of our employees from OpenAI and Anthropic and Google. This is why I am so locked in, Jonathan, to Apple, Silicon, OpenClaw, Kimmy, et cetera. I know it's not as good as Claude. I know like Perplexi Computer's awesome, but I just feel like we need to own this and we must go for it.

1:02:56Now, wait, you said there were two paths. There's a second path. And actually, I'm really curious, Jonathan, for your thoughts on the second path and Jason as well. Like I didn't, I think the second path is like right now as consumers, you know, we're kind of in the like processed food era. We haven't realized yet that organic is better for us. And so right now we're in the like, oh, let's just use what's easy. It's the processed food of like Claude. We're just going to buy at the grocery store, or get it right out of the box. But like the organic, the like open version, the open source version, like what you're setting up, Jason, all of your own stack, that's like actually what's good for us to build the infrastructure of our life on.

1:03:36Like I think 10 years from now, like our entire life infrastructure, our house, our like knowledge work, our friendships. Our health data. Our health data. Yeah, it's all going to be in here. Relationships, everything. It's all going to be in here. And I think open source does not have to be synonymous with hard to use. In the past, it was hard to use because not very many developers, not very much money went into it. But now every single person can be a developer. And so every single person can go in and go edit their open source software infrastructure. And in fact, there is an argument for open being better than closed because now you can go and change the stuff, whereas before you couldn't.

1:04:12And if it's closed, then you can't go, I can't go change my cloud desktop. It does some really annoying things. I want to add like this recurring, like slightly different scheduling functionality. I can't. But Jonathan, I think one thing I'm most stuck on is open models catching up. And you said you have all this data that you sell to the model providers and one of the, that it's very specific data, enterprise workflows, coding workflows, and the open source models just don't have that kind of data. So in your view, and Jason and Karina, yours as well, how do open models actually stay caught up as you get more and more specific and private data into them?

1:04:52Great question, Kanjian. So we work with a lot of the open model builders also. So we provide them with data as well. Many of them don't have as big budgets as many of the larger frontier models, but we also work with them. and I've wondered about this a bit. I feel like there's going to be room for the trillion parameter giant world models and smaller models and it's very workflow specific. If you're building a general assistant, you probably want knowledge about the world so you can reason about all sorts of things. But if you're doing automation of an invoice to pay workflow or automate service ticket resolution or automate the job automate the workflow of an FP &A person that's doing analysis monthly, you probably don't need, you can probably be in the half a billion to 10 billion parameter regime, fine tune that model on your proprietary data, distill the human intelligence of your humans working in your enterprise into the models, automate your proprietary tool calls.

1:05:57For that, probably a smaller open, and I realize we're mixing a few things here, like open versus closed and small versus large. And open also, there's distinctions between open source, open weights, like those are also different things. And whether you can fine tune it or not, that's another dimension. But by and large, one shift that I'm seeing is in enterprise, there used to be these two camps. I'm going to call them the fine tuning camp and the no fine tuning camp. Like two years ago, it was in the enterprise, it was a lot more of the fine tuning camp where Let's take a small model. Let's fine-tune it on proprietary data.

1:06:34Maybe we do some SFT. Maybe we do some RL and build custom models. The no-fine-tuning camp is, hey, we have this giant model, and all you in the enterprise need to do is be smart about how you manage context and how you manage memory. With intelligent context and memory management, you don't have to touch the weights. The models are pretty good at in-context learning, and you can even do continual learning without touching the weights. If you're smart about how you accumulate training samples with marginal information gain, like when the model is doing something wrong and the human is error correcting it, just like Jason error corrected some of the prompts for marketing taglines, if that could be baked into memory and context, you don't need to touch the weights.

1:07:18Forget RL, let the labs do RL. You just build the infrastructure around it. And what I'm seeing is in the last year, there's been a larger shift towards the no fine-tuning camp. Take the large model, do good things with context and memory, no need to fine-tune. Not that the fine-tuning has gone away, but that's an interesting shift because I feel like two years ago, it was very much, you need big models for consumer, enterprise, small models, on-prem, sovereign AI. And even though people talk about sovereign AI and there are important problems to solve with ensuring IP stays within the compliance boundaries of the enterprise, But I'm seeing a lot more openness to using giant models with just intelligent context and memory management than there was a while back.

1:08:05So I think that we're going to have both. I think we're going to have big models and we're going to have small models. And the Frontier Labs could make big and small models. Some of them could be open, just like OpenAI does with GPT-OSS. And some of the companies like Meta could maybe. I don't know. Well, that's the weird thing is, Karina, Microsoft, Meta, and Apple basically having done nothing of substance in AI. I mean, Microsoft, of course, bet on OpenAI. Fantastic. And they have Azure. Okay. But it's not like there's a Microsoft AI product that we would all say, oh, my God.

1:08:41Carina Hong:And they have good talents. Yeah. I can't get enough of this Microsoft blank or this Meta blank. What's the Facebook blank AI thing? And then when it comes to Apple, I'm absolutely frustrated with Siri to the point at which Whisperflow is literally this little startup, Whisperflow, has made a product that is a thousand times better than Siri. And how is that possible? They've got hundreds of billions of dollars sitting there, and they can't spell your name correct, Karina. They don't know any context about you. You tell it, like, put my address in. It's like, I don't know where you live. You're like, you're my iPhone.

1:09:24I'm on you eight hours a day. You know my address.

1:09:27Carina Hong:You have Google. It's literally in Google Maps. It's on my V card. It's everywhere on this freaking phone. So how do you think about open source and then those remaining players, if you were going to want to? I will also say that probably the breakthrough about continual learning is going to happen at one of the startups. I think there are like about three that I'm tracking. I think one of them, I'm super excited. I mean, just like, you know, watching from the sidelines, it's like the dream of personal intelligence of like, indeed, you know, you don't need a huge, huge, huge proprietary model to serve your personal daily need.

1:10:00Carina Hong:I see like, you know, two or three groups just making those breakthroughs with like top talents from XAI and from other places and from OpenAI because they are not, that direction is not supported there. And I mean, at a point, Meta had a chance to have the best, you know, model, if not open model. I mean, it was Lama. And then somehow that team kind of spinned out and went to found Mistral. And that's, I think, one of the interesting things. There's so much alphas, you know, beneath the layers of management of a big, you know, organization. And perhaps back then the CEOs didn't have the tools to go direct, right?

1:10:36Carina Hong:Like you can't actually just have your AI penetrate through the layers of bureaucracy. I'm not sure they're doing it now. I hope they are, because I really do think that small teams generally with a focus that's not easily changed can achieve a lot. And we are in this sort of interesting market where new labs are founded. And there's so many new labs. And you look at the people who are joining these companies and look at sort of the valuation. It is higher than before. But personal economics wise, it still doesn't make sense for a lot of the early founding members. But they are there. They're locked in.

1:11:07Carina Hong:And because of the dream, they are seeing that this is a direction that they can pursue for as long as it takes to bring it to realization. And I think Axiom being one of the mass super intelligence, like big players here, we are seeing people with that dream that they cannot fulfill if they join any other company. So it's interesting. I think a lot of innovations are happening in the venture startup landscape. And so people say it's a bubble, but people also say, you know, if it's not a bubble, it's a moonshot. It's one or the other. And binary outcome, fail fast, that's a really great time to be doing venture investing in a way.

1:11:43Eventually, I think with Apple Silicon and this on-device intelligence would be awesome. And I like the imbued vision of just an open system. So maybe we'll create our own intelligences that amplify us so that we could have many copies of ourselves that we fully own and control. well i like the way ken june you presented it which is i don't want to rent myself back from sam altman no definitely that is like that is my personal black mirror like that's the jcal black mirror episode is me having to go to sam saying can i have myself back and they can lock you out of yourself at any time like that's terrifying the other thing is make yourself do

1:12:25Carina Hong:discoveries and like you know if they have me they can make me do math until the end of the universe it's like pantheon you know it's it's nuts uh one thing i am curious about though yeah for for jason jonathan karina like um the economic dimension of personal uh agents and open agents like i'm really struggling to figure out you know jonathan you said the open model trainers they don't have as much money to spend because they haven't locked in as much profit because they are giving their models away kind of for free and they're just charging inference. Like economically, a big question I have is how to make the personal AI sustainable.

1:13:05You know, one way is, okay, make it so consumers only prefer it, but that seems hard to kind of convince everyone like, oh, this is going to be a huge problem for you. And so, yeah, how do we sustain this? There is a roadmap for it. We saw, you know, with WordPress, which powers like a third of the web or some crazy number, but they only monetize like 1 % of that. So that is the beauty of open source is this ability to only take a little bit back and then do commercial versions of it. What I wanted OpenClaw to do was to have openclaw.org, open source, and then do openclaw.com and make it a, and I told him like, if you raise money, obviously I'll put money in if there's room.

1:13:50It would be the largest venture round ever, but he didn't want to run a company, is my understanding of it. But you would have a.com and you could have the hosted version or the version with customer support, or you can take the free version, which tons of people use the WordPress open source software. And some people like to use the hosted version. And Apple has a unique place in this ecosystem. I just spent$3 ,400 on my MacBook Pro. I just got the 14-inch one. Usually I'm a MacBook Air, but I'm like, I need a really fast or more powerful one if I'm going to be running OpenClaw or some of these things.

1:14:26And then I'm like, well, it's obvious that everybody in my company is going to just be on a Mac Studio with 512 gigs of RAM. And as an employer, giving somebody a$10 ,000 computer, a$3 ,000 computer, or a$1 ,500 computer makes no difference to any business out there. There's no business. I mean, like even a call center. If you gave a call center employee a$10 ,000 computer versus a$2 ,000 computer, does it make any difference for that business? No. It's the least cost. And my first computer, an IBM PC Jr., I think my dad spent$1 ,500 on it in 1982 or 83. That would have been three times that, four times that with inflation.

1:15:06So it would have been a$6 ,000,$7 ,000 computer. So back then, we used to spend regularly$4 ,000,$5 ,000,$6 ,000,$7 ,000 on a computer, even$10 ,000 in today's dollars. I think we go back to that. The open source models are just tremendous and can do 90%. And then the jobs, it can't. You just have some sort of a router, an intelligent router that says, hey, this query. Yeah, like route to the frontier model if needed. And so you need a layer on top We can't just use all of Claude's ecosystem. But if things are going the way we think they are, Karina,

1:15:44what amount of model are we going to need? Are we going to be so in abundance that Kimi 7.5, four or five generations from now, just feels like I can't even use this. It's like owning a Tesla. A Tesla can go zero to 60 in under three seconds. It can go 150 miles an hour. It can go three or 400 mile range. So you don't use any of those. Right. There's a speed limit. It's unnecessary to go 150 miles per hour. It's unnecessary to have 400 miles.

1:16:15Carina Hong:Yeah, that's interesting. I mean, I'm in the camp that believe that, you know, recursive self-improvement is going to come. I believe that companies working in coding, code generation, companies working in code verification and mathematics being one part of this. that's part of one end game, at least. I believe that there are a lot of problems that are unsolved. I want to understand the universe. I can't understand the universe if I don't, or I can't even understand my own brain. I mean, neuroscience is famously hard if I don't have like literally the sharpest model that will train itself, be that AI, AI scientist.

1:16:55Carina Hong:And I firmly believe in that word. So I think it's like the MIT part of me is like scientific ambition. that is that is one line and i think that's super intelligence yeah exactly as opposed to really scientific super intelligence yeah and i also think that we have a choice which is are we building super intelligence that is slop or like you know hallucinates all the time so every five times you run it you understand the universe and the other four times you understand something else and it generates like millions and tens of millions of lines of math and you're like Like, is that the answer to the universe?

1:17:32Carina Hong:I don't know. So I think we're also at a crossroad of whether humanity as a whole choose verified super intelligence or not. And Axiom is, I think, one of, if not like the one company building verified super intelligence. But I also believe in like, you know, like my mom, like she doesn't necessarily get the most utility out of super intelligence. She wants to be, humans have like, you know, fulfillment, happiness, and personal intelligence will be another line. In a way, the quest of personal superintelligence, the word super is like too unnecessary here. In a way, it's about understanding and empathy.

1:18:06Carina Hong:And in a way, I think I'm pretty much, you know, drawn to the human's end pitch of, you know, are you going to build AI that maximizes human welfare in a way for those users? And that's very interesting as well. But I think it's like market forces is going to make recursive self-improvement happen. then market forces is going to mean winner take all, close proprietary models with large amount of funding is going to accelerate until the end of the time and the other ones will be left behind. But the thing that can counter and kind of balance out market force is ideology. There are people where I'm like, I'm just, I'm the best AI talent you can find.

1:18:44Carina Hong:I'm just not gonna work in a company that like require me to give those privacy because I don't know, I have this freedom ideology. Even if I'm in a corporate setup, like this person still expected, then you maybe a ceo will have a choice of okay am i gonna you know employ these moonshot talents that can like deliver the next gpt or am i gonna you know employ the the the less you know shiny talents who will give access and will probably we have an example karina karkathy is releasing some of the most fascinating and applicable you know repos on github that are having non-developers be like, yeah, I'd like to make an LLM myself.

1:19:27And he's like, yeah, here's how to do it. Here's a YouTube video. Here's how you can make your own LLM. Here's how you train it. That's kind of mind-blowing. You only need one Carpathie for every hundred people who are like, I need open AI stock or anthropic stock and need to make my$20 million, right?

1:19:44Carina Hong:But in a way, it's interesting is that your skill level and your economic sort of, you know, circumstance determines whether you have the right to buy your ideology. This is, I think, something that's interesting. It's like, you know, someone who is not the capacity level. I mean, it's just a very interesting end game in my view for that. It's like, what's going to counterbalance? Yeah, Jonathan, we have a term for it. It's called FU money. You get enough FU money, you're just going to do what you want. It seems like Carpathie could work everywhere, anywhere, be the co-founder of any company at any time.

1:20:21And he's just posting interesting stuff to X all day long and get up. It was super fascinating. And like, you know, sometimes you're a parent, you need to put food on the table, you know, and that's your morality. I have to feed these kids, you know, and maybe you got to steal a loaf of bread along the way. Like it is a super fascinating time. If abundance happens, maybe people are just like, I want to only work on things that are world positive. Yeah. interesting. People can choose that.

1:20:47Carina Hong:People moving from Kupin, Andrew, and Saurvic during that one weekend too. Yeah, totally. I also think we can have personal super intelligence for what it's worth, not just relegating humans to having personal intelligence and emotions. Just like today, we are far more advanced in what we think about, work on than we were 300 years ago. That's right. And I feel super optimistic about what's ahead. I feel like we are going to head to idea to company in one prompt. It may not happen in three years, but over the next decade. And I feel like we are so trained in thinking of the Andy Grove style. A human can have five to eight direct reports.

1:21:26I think the future, a human might run five to eight companies. And maybe what the human does is put their tax ID in, make sure everything does all the compliance stuff. And I think the nature of a job is also going to change. Humans will, I think, transition from doing the work to like verifying the work of agents doing the work. And maybe people will all have multiple fractional jobs where you could be running different companies, working at different companies. I think the nine to five or like 24 seven, like one company thing may change. One question I have for you, Karina, is it's so inspiring to work on solving or on building like math superintelligence.

1:22:09What does the world look like where there is a math superintelligence? Yeah, that's interesting.

1:22:15Carina Hong:I think so the dream that that we really have is you have like a billion AI goals working for you. AI gods? Goals, like the mathematician, the child quality goals. So you will have like, you know, all these abundant outlier reasoning capability. I think humanity is pretty sad that we have been reasoning bound for some time. I mean, you have Gaolua who died at the age of 21, 22 out of that famous like romantic duo. And then group theory kind of like got set back for like decades. Like one person died and then entire humanity got set back for decades. And similar story with like Ramanujan, right?

1:22:54Carina Hong:Like, you know, malnutrition. And then, you know, you stop having these intuitions in number theories, those like magical formulas that other people build on. Still today, mathematicians are trying to decipher his lost notebook. Now we want to have AI doing mathematical discovery. And then the sort of time span from a mathematical breakthrough to like apply science and actual market sort of advances shorten from like 300 years to like three days. like I want I want that word and in a way that really complements the whole coding advances because those are like try and error and this is first principle and you kind of need both and you also need first principle to now verify verify trainer which I think is a really beautiful part when we started out we didn't know there's this verification angle so now it's beautiful that math needs code and code needs math and it's it's not one direction and so that's really the future that we are building.

1:23:49Carina Hong:And in a way, it will be very interesting to see like how all these sort of company operations are being kind of coded up in the software stack. How much of the world can be explained by theoretical math? I mean, theory of deep learning is a field that hasn't really been mature because most of the theory people are able to analyze about deep neural networks are like assuming one layer, assuming a linear transformer. And we still don't understand anything. I mean, the fact that transformers are not understood and we don't know the output, but we know it's doing an incredible job reasoning. That's right.

1:24:24It makes one wonder about our own brain, you know.

1:24:28Carina Hong:Yes, about our own brain. And you have, you know, like probably one in a hundred neuroscientists who come from a math major. And MIT, Yila Feet, she applied Chinese remainder theorem, something about elementary number theory, remainders, to analyze neural capacity. like how many neurons you can store in your brain and then what's spacing like and and you know it's physicists who kind of like did string theory who come into neuroscience really push forward the field of theoretical neuroscience and i just want the same to happen with ai mathematicians in every single field i love that karina i i'm gonna steal that line of humanity is reasoning bound that's a neat way to think of it you're right and your point about like it'll be nice to have like a million guvs.

1:25:14Maybe you could call your intelligence super normal intelligence or ultra normal. Very bad. I love that. All right. Listen, this has been an amazing episode. Jonathan from Turing, thank you for coming. You're hiring, I'm assuming. Where can people find out more about Turing? Turing.com. And if you want to work on research to help advance the models for coding, SWE, Enterprise, join us. Karina, tell us a little bit about where people can find you more about Axiom.

1:25:43Carina Hong:Go to axiommath.ai. Recently two released. One is open source, so you can run on your computer to discover interesting graphs, like Turing graphs. And it's called Explorer. The other one is also free hosted open service called Axo. If you want to run large mass computer program, Axo is your go-to infrastructure, and community has been using it. And Kanjun Q and Bue, where can they find more? and what should they learn about? Yeah, you can go to imbue.com. We just open sourced a product last week for you to run a fleet of parallel coding agents so you can make your engineering team much more automated.

1:26:21And also it's not just specific to code. So you can actually build business processes into it. We do that. We have a bunch of people who do that. And we are definitely hiring. If you want to build products from zero to one, that's where we run our whole company that way. All right. And we'll see you all next time on This Week in AI. Bye-bye.