Every AI Founder Should Be Asking These Questions

0:00Welcome to this talk. I'm extremely confused. I think maybe more confused than I've ever been in my entire life. probably definitely more confused than I've ever been in my entire life. And I want to talk about that because I think, I don't know, when you're confused, that's the start of something interesting. If you're trained as a scientist, that's like the thing you want to pay most attention to is that moment where you're confused. Like, oh, this is interesting. I don't know what's about to happen. I've been in technology for like my whole life. And I feel like I've always had this advantage where it's like, oh, I understand what's happening.

0:32I actually kind of think I know what's going to happen over the next five or 10 years. And I've used that to my advantage. Like I've planned my career around it. I've founded companies around it. You know, I've kind of gotten ahead of the trend and gotten involved at the right time in a lot of different places. And that's like really been great for me. I no longer feel that way though. Like I can't see five years into the future. I can see like three weeks or less. And I really just don't know what's gonna happen. So I wanna have a series of questions that hopefully help me figure out what's going on.

1:07Maybe it'll help you all figure out what's going on. Maybe not. Not all questions are useful. But I think asking good questions is extremely important to running a startup, running a research team, running your life, whatever you're doing, I think we all need to stop sometimes and just ask questions. But I think the moment in time that we're going through right now with AI is extremely challenging and fast. And if there was ever a time to stop and ask questions, it's probably right now. All views are my own, obviously. I think we're required to say that. My day job is running an alignment research team at Anthropic.

1:41I've also been through YC. I've done multiple startups in my life. So maybe my perspective on AI plus startups is useful for all if you're thinking about startup life or AI life. Okay, that's enough of a preamble. Let's jump in. This is sort of like the main question that I want to ask. Everything's changing. How should that impact everything about my life? Honestly, like, should you even start a startup, like, is a big question. But let's assume that you're starting a startup or you're running a startup already. How should it impact your strategy? How should it impact your product? How should it impact how you're building your team?

2:15Like, these are big, open questions. And I think AI is probably going to change how you answer them today and probably differently tomorrow. And there's like this kind of paradox that I've seen throughout my career running startups and talking to startup founders. Everyone always tells you focus is everything. Like that's the advantage that startups have. Focus, focus, focus. Big companies can't focus. That's why you can out-compete them and run circles around them. But despite the fact that focus is everything, the other truth of running a startup is you have to focus on everything. Like it's your job to think about hiring and fundraising and product and strategy and go to market and just like everything.

2:51Right. and then all of a sudden in the middle of like a product launch someone on your team quits and someone else threatens to quit and it's just like it's madness and it's happening all the time. This is just an extreme paradox but I think it kind of makes founders suitable to this biggest question that we're all facing as a society which is what's going to happen with AI and what should we be doing because founders are the people that have to just answer every single question always so I think it's a good positioning. There's been sort of like common quote-unquote common just like in the last six months, best practices, like what do you think about AI for your product?

3:25And people say like, oh, you should like think about what happens over the next six months. Think about what the next foundation models are gonna be able to do and make sure you're planning your product, anticipating what those capabilities are gonna do. Don't like plan for the capabilities of today. I think it's like extremely valid advice. You should definitely take it seriously. But I wanna like up the ante a little bit and say, actually, you should be planning two years in advance because it's extremely likely that we will have AGI in the next few years. Maybe it's not two, maybe it's three, right?

3:52But I think you should be planning your company and your strategy around this fact, right? And there's extreme uncertainty, right? So don't take it too seriously. Don't have literally a two-year plan. But if you're not thinking a little bit about how this is going to change everything from hiring to marketing to go to market, et cetera, I don't think you're doing your job as a founder. Okay, so that's really the theme here today. I want to go through a series of questions and take this lens of AGI arriving and also just take the lens of what's changing near term in AI over the next six months.

4:24I think there's been a lot of conversations around like, oh, actually, the impact of AI is going to be slower than we think. And the reason is that big companies suck and they have to like they take a lot of time to buy things. They don't realize the trend and then the enterprise sales cycle is really slow. So all the Fortune 500 companies are just going to take years and years to digest all the SaaS products that you all might want to be building. And I think that's actually extremely nearsighted. Because what's going to happen, I think, open question, is the buy side, the enterprises, they're going to get armed with AGI or strong agents over the next couple of years as well.

4:58And that's not just via the SaaS products that they might be building. It'll just be natively inside. Their teams are going to be using the next versions of LLMs to make buying decisions and to figure out how to accelerate their adoption cycle. The force of AI is not just on this product revolution that the startups are building. It's also on the buy side. And I think this is an interesting, weird thing about AI is that it's going to rise. The water rises and all ships rise with it. It's not just the startups. The incumbents benefit from AI too. And I think we're seeing that in other places too, where large enterprises sometimes aren't even going out and buying software from SaaS providers anymore.

5:36They're like, I can just like throw two people at Cloud Code and they'll build it and it'll be dedicated to the capabilities that I need for my organization. So buy side is going to be evolving really quickly. And I think it's an open question. What does that mean? Right. I think another angle on this is, you know, when you have AI powered outbound sales, we think about what that is going to do to the marketplace. but there's going to be this in this buy side too of like receiving those sale calls from ais and trying to parse what's going on so the dynamics aren't really clear how it's going to play out this is a kind of related question like is software going to fully commoditize is it even going to make sense to run a sas provider in like two years or three years or is it like going to be the case that enterprises really do just build all their software in-house because it's just one you know one prompt to cloud code the next gen version of cloud code right and actually you just need in-house product managers and they're going to do everything for you like that's one real outcome that could happen to consumer side too like maybe consumers eventually are just like not downloading apps anymore they're just building apps on demand for themselves they don't even think about it that way like they don't even think about them as apps anymore it's just yeah i want my phone to do something for me i ask it and yeah it made an app for me sure um what's the point of downloading an app at this point so that's one outcome but i think another outcome actually is the opposite it's like maybe all of this automation on on generating code makes it easier to just raise the quality bar extremely high and sure you can make like the equivalent of today's apps very easily just by prompting but can you make an exceptional app tomorrow uh the equivalent of a great team that's working with ai uh to raise the bar i don't actually know the answer to this question but i think it might be different depending on the vertical um and you should be thinking about this for your product i think it's an interesting question too like if you can write software on demand do you even make apps anymore or is it is it really on demand right like Like we're taking cloud code and things like it to ahead of time figure out what a user needs and then build as much software as we need to support all those use cases.

7:30But if you can do code on demand, why not do it on demand? You know, on the fly, here's this user. They're doing something in your app potentially. And the app realizes that the app can't support what they want. And on demand, you generate code for that user. I think this is a really interesting pattern if you can make it work. But it raises all sorts of issues around like trust, right? It's one thing to do generative UI, where maybe you're changing the shape of the interface for the user. But real new behaviors would require going down to the database level or the backend. And if you want your AI to be able to do that on demand, you better trust that that AI can do its job.

8:03And obviously right now, AIs are not trustable enough to make that happen. So I think trust is going to be a big theme over how these different questions play out. Similar question, what should UI look like? I think people talk about generative UI. It hasn't really happened yet, but I'm bullish to see when it does happen. Is an on-demand UI the right interface or is there something even more different that we haven't thought about? I think about this specifically in the context of multimodality, weaving together auditory and images and video and text. And what's the right input back from the user?

8:33As a user, sometimes I want to speak. Sometimes I want to use touch interfaces. It's really contextual depending on whether I'm in a crowded area, et cetera. I think you want to meet the user where they are. What's the easiest way for them to interface with your product right now? That's the question you're thinking about. And I think all this ties into this big question they have, which is like, we have all these products today that have great distribution and people realize that we need to insert AI into it. All the major players are doing this, right? Slapping like a chatbot on, slapping on some like agentic behavior or whatever, right?

9:03They're like retrofitting their products to try to enable something with AI, right? But that's like trying to like sprinkle pixie dust on something. It's natural to think it's better to build it a new product from the ground up that's AI native. like that's that's like my startup mentality like new technology revolution you need to start from scratch if you want to build this right but that might not be true it actually might be the case that retrofitting existing products with the benefit of distribution wins and i think again it might not it might not be universal this might be vertical by vertical but i think this is a really important question that you should be asking yourself and like don't just have an opinion like figure out the causal mechanisms that allow you to validate your hypothesis because i think these types of questions will make or break different products.

9:46Okay, so that's a lot about like products, product strategy, but I think every part of a startup is like extremely important. The most important thing is your culture, it's your team. Will team sizes get even smaller? I think like that's like the default that people are assuming. But I think similar to products where like you can retrofit or you can build from scratch, I think there's going to be this question which is are like AI native teams that were built from the scratch to be AI native, are they going to have some advantage over large companies that are like downsizing and finding ways to make themselves more efficient with AI, right?

10:17Are there like team patterns where you kind of like just operate differently if you started from an AI native place? And actually that might be different every 6, 12 or 18 months, right? Because the capabilities of AI are changing. So like an AI native company today might be different than what an AI native company tomorrow looks like in 12 months. You might be outdated if you're not like thinking about how to retrofit yourself. Like I said, I think trust is going to be extremely important right so like how does the security model change like i mentioned this already in the case of on-demand code where you want your llm to be able to go all the way back down to the database layer and do something on demand for that customer for that consumer but you can't do that if you can't trust the controls that you have in place and if you can't trust the capabilities of the model to do the right thing right i think about this also in the context of like assistance like the whole point of an assistant is to be useful to the to the consumer to that user but we have all these walled gardens and you're starting to see like maybe you need different agents in different settings right but that's not what you want as a user you want one agent for all of your things right so me i want my personal agent and my professional agent at work to be able to work together right that raises all sorts of concerns like there's stuff that i do on my personal time or you do on your personal time that maybe you don't want your employer to know about right so how do you make sure that the information is segregated while still allowing those agents to collaborate in the right way.

11:36Those are hard questions. Can you do it, right? Can you get to the same level of usefulness or do you have to compromise somehow on some of these security aspects? A deeper question about agents is, you know, we think about alignment as can we trust the AI, but the truth is even if you have a perfectly aligned, you know, quote-unquote, like intent aligned model, it's going to be used by a corporation or a startup to build an agent for that user? And then the question is, can you trust the startup that's building the agent, right? Is this agent that they've developed for you actually acting on your behalf?

12:13If this is like an ad-based company and you're using this agent to search for like a new brand of shoes or whatever, right? Like, is it going to be biased? Is it going to be like pumping you towards one direction? That's not what you want as the user, right? But it might be what you want as the corporation. And then how do we start even asking these questions, right? I think going back to the previous slide around like personal versus professional agents, I really care if I'm using that agent, I really care that it's operating on my behalf, right? If it's like secretly operating on behalf of my company and it's like optimizing things in my personal life that's better for the company than me, that's like extremely scary.

12:49And I think it gets more scary the more capable the models get. So I think it's important to start thinking about trust, right? Like how do we, how do we instill trust not just in the models, but in the agents and in the companies that are building these agents, right? And I think ultimately what that really comes down to is trust in you all, or trust in the companies that you're building, right? How can a user trust this company that's building this product, especially already today we have these extremely small teams. Tomorrow the teams might be even smaller. You might have these semi-automated teams almost.

13:20How can you trust that team? You might say, well, I don't know, we trust companies today. One of the core reasons we trust companies today is because they're composed of a diversity of people. You can trust to some extent that if there's reasonable culture at a company, that if the company decides, if the CEO decides to do something bad, that someone in the company is going to raise their hand and say, no, I don't support this. I'm going to whistleblow. I'm going to leak this. I'm going to quit. I'm going to take a bunch of people with me. I'm pissed. I hate this. And without the people supporting the company, the company doesn't have a product.

13:54But in a semi-automated world, that's no longer true. And it could be the fact that a single person could make a decision that changes the entire impact of a product. And there's no single person that might be aware of that except themselves. It gets extremely easy for bad actors to do bad things. And if you've followed along the history of Silicon Valley or the history of humanity, the truth is a huge majority of people are misaligned, right? Especially when money is on the line. So I think this is a really important question. And the truth is like, we already think about this. Like large enterprises, for example, already distrust startups partially for this reason.

14:29There's a lot of reasons why large enterprises distrust startups. One of them is they're worried they're going to go out of business tomorrow, right? But it's just a lot easier for a small startup to do the wrong thing compared to a big company, right? That's part of what makes small startups successful sometimes. So I think this is already like on the top of minds of enterprises, but it'll increasingly be on the top of minds of everyday people. So that kind of leads me to this next question, which is, well, how do we instill trust? Like what does a new set of guardrails need to look like? If you don't have all these human guardrails inside of your company where you've worked really hard to build great culture and build a collection of people who care about doing the right thing, if that's not the thing that makes a company ethical anymore, how else can we make sure that there are guardrails?

15:10What does that look like? A lot of people have been throwing around ideas on this. There's this notion of AI-powered auditing. What should an audit look like in an AGR world? There's this huge advantage that AI has over humans when it comes to bringing comfort to an audit, which is that they can be less biased and they can also have no memory. If you, for an example, right, if you agree as a company to let an AI audit you and you can say, hey, if you don't find any malfeasance, if after your audit you've decided that we didn't do anything wrong as we claimed, maybe you have like a public mission statement, like the AI is like confirming, then the AI deletes itself, right?

15:48All of its notes, et cetera, get deleted. And that's a big advantage over having human auditors where they could take information with them, right? There's already auditing happening today, right? For legal reasons, for financial reasons, or you want to be like certified organic or whatever it is, right? But you have to let people come into your company and audit you and there's danger there, right? They might take with them like IP or they might find sensitive things and then like, you know, which were unrelated to the thing you wanted to get audited and all these bad things can happen, right?

16:15So there's like this potential, I think, for AI to give us a much stronger auditing system that can be part of how we build trust, how we let people build trust in us and the companies and products that we're building. So that kind of leads me to this next question, which is, should we be doing that? Like, are there some other ways that you plan on having, on building trust with your users? Or should we be doing this? Should we be making like, not just public commitments, companies make public commitments all the time. Like, oh, we care about this. We care about open source. We care about doing the right thing for the user.

16:47But are you actually willing to like make that a binding statement? Are you really, really willing to like stick your neck out and say, yeah, not only do I say that this is what I believe to be valuable, I'm willing to commit to an ongoing audit from some neutral arbiter, from some neutral AI powered system that will come in and inspect every single thing that happened in my company, every Slack message, everything, and ensure that actually we are making decisions that abide by the mission statement of the company. That would be something that might have teeth. Maybe this isn't the right way to do it, but I think things like this are going to be possible soon.

17:20They're not possible today. And they might be the flavor of things that we need to start building trust once we're in this world where we've lost a lot of trust. Related question around alignment. So I think a lot about alignment. And I think there's this question of like, what parts of alignment do we have to solve? One reason we're working on alignment is because of the control issue. We want to make sure AIs stay under the control of humans. But I think there's this extremely high pressure question for the next 12 months, which is what parts of alignment do we have to solve just to make these models more economically viable, right?

17:52Just to make sure that the agents that all the startups are building, as their horizons get longer and longer, that we can actually trust those agents are like not going off the rails, right? And if you use Claude code today, and it works like five minutes at a time for you, that's like one thing, because you're going to review a lot of what Claude does. But if you're going to trust an LLM to work for a day at a time or a week at a time before you intervene, you better have some degree of certainty that it's not going completely off the rails, Right. So I think this is like a huge open question.

18:19I'm actually really positive and bullish that there is this economic pressure in a good way to make progress on alignment because long horizon agents require it. But I think it's an open question how much and what aspects of alignment have to be solved for this. Changing topics a little bit. I think this is like a question that we all like maybe think has been answered, which is, is there a set of data that can give you an advantage? And if you go back in time just a handful of years before frontier models, before LLMs, the assumption, not just the assumption, the fact was that custom data mattered.

18:49If you wanted to build an AI startup or if you're an enterprise and you're trying to deploy AI, you had this massive advantage if you had a massive data set that was custom to your need. And that was actually the only way to get useful AIs more than a handful of years ago was by training models on your custom data set. And then very quickly, what we saw was LLMs got extremely powerful and extremely general. And actually, it was just better to use the general LLM than it was to train or even fine-tune on your custom data, right? What I think is true is, maybe it's true, this is my open question, is there might be industries where that's not the case, where actually AI is maybe not great.

19:24An example might be like material science. Can an LLM do really good material science? Does a company that specializes in material science, that has decades of data, can they do better? And I think potentially, yes, right? Like LLMs are great at everything that they've found on the internet, but are they great at all this like tacit knowledge that's locked up in companies that actually hasn't bled out? I think about like TSMC or ASML, right? Like they make these multi-billion dollar bets and they do an incredibly good job of keeping all that tacit knowledge they build in-house. It doesn't leak out.

19:54Frontier LLMs do not know how to build a cutting edge semiconductor fab. That's an important fact, actually. And I think if you're a startup thinking about where there's a defensible position, and I'll touch on this again in a few slides, I think that's an important question to keep in mind. Okay, different tack tiers. You're probably all aware of capacity issues. Everyone's trying to scale extremely quickly. I think there's demand from consumers, from your consumers potentially, from your startups, to scale extremely fast, right? Maybe we want to scale like 100x over the next couple of years.

20:27That's faster than we can scale GPU production. So what are we going to do? And I think there's a lot of open questions around like, does fine tuning actually matter? I think a lot of people have abandoned fine tuning and said, actually, I'm just going to do better context management. Or is there like a role to play in like better routers between which model, small versus large, etc. So I think this is like a place where if you're interested in the technical details, you can have a competitive advantage, at least for the next year or two, because capacity is really going to matter. Right. You know, often from a product perspective, I like to say, make it great, then make it scale.

21:00if you're already starting to work on the make it scale part, this is really important to you. And this can be some of the technical moat that you can build that gets you ahead of the competitors. Getting ahead of the competitors, though, is like it's a rat race. So at some point, the models will get better, the capacity problems will improve, and then that advantage might go away. So you need to have a better answer to this question of like, what's your moat? How are you going to stay ahead? What makes a durable advantage? in a post-AGI world, right? In two years or three years, if I can just prompt, you know, Claude 7 or GPT-7 to just replicate your startup, what's your advantage gonna be, right?

21:40And if I'm a megacorp and I have more money than you and I can throw more tokens at that, like, are you really going to have a durable advantage or are you just gonna get clobbered by megacorp? I think it's like a real question that is extremely serious. Even before AGI, I like to work on hard problems. Like, that's the moat for me. Everyone has a different type of mode. Some people are great at marketing or whatever, right? I like solving hard problems. But I think about what does it mean to be a hard problem? What's going to be hard in a post-AGI world? And I think a lot of things, actually, like TSMC, like ASML, those are hard problems that eventually will get easy with robotics, etc.

22:18But robotics are lagging behind. So I think there's these hard problems that will exist even in two years. And if you're willing to go after hard problems, like if you have the guts to do that, then you can have a massive competitive advantage, right? So like, what is that set of hard problems? You know, infrastructure and energy and manufacturing and chips. What else? Like, those are the things that are top of mind for me. But like, what are your answers to this question? Like, what's still going to be hard and worth doing? I think about this question a lot too. Like, is there an intelligence ceiling to what you need for various different tasks, right?

22:50Like, you know, if you look at ImageGen, for example, you know, just recently with Veo3, with VideoGen. It's like, I can, I finally am getting fooled by videos. And like my Instagram feed is starting to get full up of like copybaras singing songs and shit. And, and I'm like, actually, this is great. I love these videos. That wasn't true three months ago. Right. But can it get even better? Or is there like for a specific task for a specific use case? Is there like some max where it's like, yeah, that's, it's good enough. We've saturated, right? Like that's the best you can get at writing a poem.

23:22That's the best you can get at writing a get diff for a PR, that's an important question because if there is a ceiling, then the commoditization for that task is going to hit much sooner, right? You actually can't stay on the edge longer by moving to the next model, figuring out how to prompt the next model. The task saturates, right? So the commoditization pressure will be even more extreme. So I think it's important to think about this. And again, I think it's going to depend on the task and the vertical. Like some things, maybe there won't be a ceiling. Some things, maybe there will. This is changing tack a little bit here, but is there going to be a need for neutrality?

23:57Since the dawn of chatbots two or three years ago or whatever, people have been complaining about refusals as an example. Oh, I asked the model to do this and it refused. If we end up relying on these models, that's a massive question for society, right? That there's going to be a handful of corporations that get to decide what is okay and not okay for an AI to do for you. And if we start relying on AI to do everything, then those companies become arbiters of what gets built, right? That's extremely important. So is there going to need to be a notion of neutrality? Like we have today with some forms of infrastructure, right?