20VC: Sam Altman on The Trajectory of Model Capability Improvements: Will Scaling Laws Continue | Semi-Conductor Supply Chains | What Startups Will be Steamrolled by OpenAI and Where is Opportunity

0:00We are going to try our hardest and believe we will succeed at making our models better and better and better. If you are building a business that patches some current small shortcomings, if we do our job right, that will not be as important in the future. We believe that we are on a pretty, a quite steep trajectory of improvement and that the current shortcomings of the models today will just be taken care of by future generations. I encourage people to be aligned with that. This is 20VC with me hiring. Stabbing some water discussion we have few to date. I was very honored to be asked interview Sam Altman at OpenAI's Dev Day in London, and our episode today is the exclusive of that discussion.

0:40For those that have been living under a rock, Sam Altman is the CEO of OpenAI, one of the most important companies in history. OpenAI is on a mission to ensure that artificial general intelligence benefits all of humanity. Prior to OpenAI, Sam was the president of Y Combinator and an angel investor in Stripe, Airbnb, Reddit and Instacart. If you're inspired by his insights, you'll love the Sam Altman book summary collection on the Blinkist app. With Blinkist, you can access first class summaries of his favorite books and 7 .5 ,000 more to read and to listen to in just 15 minutes. Just to mention some, zero to one by Peter Teal, or the beginning of Infinity by David be a doctor.

1:23Being recommended by the New York Times and Apple CEO Tim Cook, it's no surprise 82 % of Blinkist users see themselves as self optimizers and 65 % say it's essential for business and career growth. And speaking of growth, over 32 million users have been empowered by Blinkist since 2012. As a 20 VC listener, hey, you get a 25 % discount on Blinkist. That's B -L -I -N -K -I -S -T. Just visit Blinkist .com forward slash 20VC to claim your discount and transform the way you learn. And while you're optimizing your learning, let's also optimize your finances with Brex. The financial stack founders can bank on.

2:07Brex knows that nearly 40 % of startups fail because they run out of cash, so they built a banking experience that takes every dollar further. It's such a difference from traditional banking options that leave your cash, sitting idle while chipping away at it with fees. To help you protect your cash and extend your runway, Brackets combine the best things about checking, Treasury and FDIC insurance in one powerhouse account. You can send and receive money worldwide at lightning speed. You can get 20X the standard FDIC protection through program banks and you can earn industry 3 -leading yield from your first dollar while still being able to access your funds any time.

2:47Bracks is a top choice for startups. In fact, hey, it's used by one in every three startups in the US. Just check them out now, bracks .com -4 -slash -dodups. And talking about building trust, Secureframe provides incredible levels of trust to your customers through automation. Secureframe empowers businesses to build trust with customers by simplifying information security and compliance through AI and automation. Thousands of fast growing businesses, including Nasdaq, Angel List, Doodle and Coda, trust Secure Frame to expedite their compliance journey for global security and privacy standards, such as SOC2, ISO27001, HIPER, GDPR and more, but by top tier investors and corporations, such as Google and Client Perkins.

3:34The company is among Forbes's list of top 100 startup employers for 2023 and business insiders list of the 34 most promising AI startups for 2023. Learn more today at Secureframe .com. It is a must. You have now arrived at your destination. Hello everyone, welcome to OpenAI Dev Day. I am Harry Stebings of 20VC and I am very, very excited to interview Sam Altman. Sam, thank you for letting me do this today with you. Thanks for doing it. I want to start by kind of diving. We had a lot of fantastic questions from the audience across a number of different areas. And I want to start with actually the question of when we look forward, is the future of open AI more models like O1 or is it larger models that we would maybe have expected evolved?

4:24How do we think about that? I mean, we want to make things better across the board, but this direction of reasoning models is of particular importance to us. I hope recently we'll unlock a lot of the things that we've been waiting years to do. The ability for models like this to, for example, contribute to new science, help write a lot more, very difficult code that I think can drive things forward to a significant degree. So you should expect rapid improvement in the O -series of models and it's of great strategic importance to us. When we look forward to OpenAI's future plans, how do you think about developing no code tools for non -technical founders to build and scale AI apps.

5:02How do you think about that? It'll get there for sure. The first step will be tools that make people who know how to code well, more productive, but eventually I think we can offer really high quality no code tools and already there's some out there that makes sense, but you can't sort of in a no code way say I have like a full start if I want to build. That's going to take a while. When we look at where we are in the stat today, open AI sits in a certain place. How far up the stack is OpenAI going to go? Is just spending a lot of time shooting your access system? Is this a waste of time? Because OpenAI ultimately thinks I'll own this part of the application layer?

5:37And how do you answer a founder who has that question? The general answer we try to give, we are going to try our hardest and believe we will succeed at making our models better and better and better. And if you are building a business that patches some current small shortcomings, If we do our job right, then that will not be as important in the future. If on the other hand, you build a company that benefits from the model getting better and better. If an Oracle told you today that O4 was going to be just absolutely incredible and do all of these things that right now feel impossible and you were happy about that, then maybe we're wrong, but at least that's what we're going for.

6:19And if instead you say, okay, there's this area where there are many, but you pick one of the many areas where O1 preview underperforms and someone patch this and just barely get it to work, then you're sort of assuming that the next turn of the model crank won't be as good as we think it will be. That is the general philosophical message we try to get out to startups like we believe that we are on a pretty quite steep trajectory of improvement and that the current shortcomings of the models today will just be taken care of by future generations. I encourage people to be in line with that. We did an interview before with Brad.

6:54Sorry, it's not quite on schedule, but I think the show's always been successful when we kind of go a little bit off -schedule. These go totally off. But there was this brilliant kind of meme that came out of it. You said wearing this 20 VC jump, which is incredibly proud of me. For certain segments like the one you mentioned there, there would be the potential to steamroll. If you're thinking it's a founder -stay building, Where is OpenAI going to potentially come and steamroll versus where they're not? Also for me as an investor, trying to invest in opportunities that aren't going to get damaged.

7:22How should founders and me as an investor think about that? There will be many trillions of dollars of market cap that gets created, new market cap that gets created by using AI to build products and services that were either impossible or quite impractical before. There is this one set of areas where we're going to try to make your relevant, which is, you know, we just want the models to be really, really good, such that you don't have to like fight so hard to get them to do what you want to do. But all this other stuff, which is building these incredible products and services on top of this new technology, we think that just gets better and better.

7:58One of the surprises to me early on, and this is no longer the case, but in like the GPT 3 .5 days, it felt like 95 % of startup, something like that, wanted to bet against the models getting way better. And they were doing these things where we can already see GBD4 coming and we're like, man, it's going to be so good. It's not going to have these problems. If you're building a tool just to get around this one short coming of the model, that's going to become less and less relevant. And we forget how bad the models were a couple of years ago. It hasn't been that long on the calendar. But there were just a lot of things.

8:32And so it seemed like these good areas to build a thing to plug a hole rather than to build something to go deliver the great AI tutor or the great AI medical advisor or whatever. And so I felt like 95 % of people that were like betting against the models getting better, 5 % of people betting for the models getting better. I think that's now reversed. I think people have like internalized the rate of improvement and have heard us on what we intend to do. It no longer seems to be such an issue, but it was something we used to fret about a lot, because we kind of, we saw it was going to happen to all of these very hardworking people.

9:06Well, you said about the trillions of dollars of value to be created there, and I promise we will return to these brilliant questions. I'm sure you saw, I'm not sure if you saw, but Massa sit on stage and say, we will have not, I'm not going to do an accent. It's, my accent's terrible, but there will be nine trillion dollars of value created every single year, which will offset the nine trillion dollar capex that he thought would be needed. I'm just intrigued. How did you think about that when you saw that? How do you reflect on that? I think like if we can get it right with an order of magnitude, that's good enough for now.

9:37There's clearly going to be a lot of cap expent and clearly a lot of value created. This happens with every other mega technological revolution of which this is clearly one. But you know, like next year will be a big push for us into these next generation systems. You talked about when there could be like a no -code software agent. I don't know how long that's going to take, but if we use that as an example and imagine forward towards it, think about how much economic value gets unlocked for the world. if anybody can just describe a whole company's worth of software that they want. This is a ways away, obviously.

10:10But when we get there and have it happen, think about how difficult and how expensive that is now. Think about how much value it creates if you keep the same amount of value, but make it wildly more accessible and less expensive. That's really powerful. And I think we'll see many other examples like that. I mentioned earlier, like healthcare and education, but those are two that are both trillions of dollars of value to the world to get right. And if AI can really, really, truly enable this to happen in a different way than it has before, I don't think big numbers are the point, and there also the debate about whether it's 9 trillion or 1 trillion or whatever, like, smarter people than me, it takes to figure that out.

10:45But the value creation does seem just unbelievable here. We're going to get to agents in terms of kind of how that values delivered. In terms of the like the delivery mechanism for which it's valid, open source is an incredibly prominent method through which it could be. How do you think about the role of open source in the future of AI and how does internal discussions that like for you when the question comes Should we open source any models or some models? There's clearly a really important place in the ecosystem for open source models There's also really good open source models that now exist I think there's also a place for like nicely offered well -integrated services and APIs And I think it makes sense that all of this stuff is an offer and people will pick what what works for them as a delivery mechanism and we have the open source as end -proptor customers in a way to deliver that, we can have agents.

11:33I think there's a lot of semantic confusion around what an agent is. How do you think about the definition of agents today? What is an agent to you? This is like my off -the -cuff answer. It's not well considered, but something that I can give a long duration task to and provide minimal supervision during execution for. What do you think people think about agents that actually they get wrong? Well, it's more like, I don't think any of us yet have an intuition for what this is going to be like. You know, we're all gesturing at something that seems important. Maybe I can give the following example.

12:07When people talk about an AI agent acting on their behalf, the main example they seem to give fairly consistently is, you know, you can like ask the agent to go book you a restaurant reservation and either it can like use open table or it can like call the restaurant. Okay, sure, that's like a mildly annoying thing to have to do and it maybe like saves you some work. One of the things that I think is interesting is a world where you can just do things that you wouldn't or couldn't do as a human. So what if instead of calling one restaurant to make a reservation, my agent would call me like 300 and figure out which one had the best food for me or some special thing available or whatever.

12:46And then you would say, well, that's like really annoying if your agent is calling 300 restaurants. But if it's an agent answering each of those 300 places, then no problem. And it can be this massively parallel thing that a human can't do. So that's like a trivial example, but there are these limitations to human bandwidth that maybe these agents won't have. The category I think though is more interesting is not the one that people normally talk about where you have this thing calling restaurants for you. But something that's more like a really smart senior coworker where you can collaborate I'm a project with and the agent can go do like a two -day task or two -week task really well and Pinging it when it has questions, but come back to you with like a great work product This is fundamentally changed the way that SAS is priced and normally it's on a per se basis But now you're actually kind of replacing labor so to speak.

13:36How do you think about the future of Pricing with that in mind when you are such a cool part of an enterprise workforce? I'll speculate here for fun, but we really have no idea. I mean, I could imagine a world where you can say like I want One GPU or 10 GPUs or 100 GPUs to just be like, churning on my problems all the time. You're not like paying for seed or even per agent, but you're like, it's priced based off the amount of compute that's like working on your problems all the time. Do we need to build specific models for agentsic use? Or do we not? How do you think about that? There's a huge amount of infrastructure in scaffolding to build for sure, but I think O1 points the way to a model that is capable of doing great agentic tasks.

14:17on the model side. Sam, everyone says that models are depreciating assets, the commoditization of models is so rife. How do you respond and think about that? And when you think about the increasing capital intensity to train models, are we actually seeing the reversion of that where it requires so much money that actually very few people can do it? It's definitely true that they're depreciating assets. This thing that they're not, though, worth as much as they cost to train, that seems totally wrong. To say nothing of the fact that there's like a there's a positive compounding effect as you learn to train these models You get better at training the next one But the actual like revenue we can make from a model.

14:56I think justifies the investment to be fair I don't think that's true for everyone and there's a lot of there are probably too many people training very similar models And if you're a little behind or if you don't have a product with the sort of normal rules of business that make that product sticky and valuable then yeah, maybe it's harder to get a return on the investment. We're very fortunate to have chat GVT and hundreds of millions of people that use our models and so even if it costs a lot, we could to like, amortize that cost across a lot of people. How do you think about how open AI models continue to differentiate over time and where you most want to focus to expand that differentiation?

15:35Reasoning is our current most important area of focus. I think this is what unlocks the next massive leap forward in value created. We'll improve them in lots of ways. We will do multimodal work. We will do other features in the models that we think are super important to the ways that people want to use these things. How do you think about reasoning and multimodal work like that? I hope it's just going to work. I mean, it obviously takes some doing to get done. But people like when they're babies and toddlers before they're good at language can still do quite complex visual reasoning. So clearly this is possible.

16:09How all vision capabilities scale with new in -reference time paradigm set by O1? Uh, without spoiling anything, I would expect rapid progress in image -based models. Going off schedule is one thing, try to tease that up, might get me in real trouble. How does open -air I make breakthroughs in terms of core reasoning? Do we need to start pushing into reinforcement learning as a pathway or other new techniques aside from the transformer? I mean, there's two questions, and there's how we do it, and then there's everyone's favorite question, which is what comes beyond transformer. How we do it is like our special sauce.

16:44It's easy. It's really easy to copy something you know works. And one of the reasons that people don't talk about about why it's so easy, is you have the conviction to know it's possible. And so after a research lab does something, even if you don't know exactly how they did it, it's, say, easy, but it's doable to go off and copy it. And you can see this in the replications of GPT -4, and I'm sure you'll see this in applications of 01. What is really hard and the thing that I'm most proud of about our culture is the repeated ability to go off and do something new and totally unproven. A lot of organizations, not I'm not talking about AI research just generally, a lot of organizations talk about the ability to do this.

17:25There are very few that do across any field. And in some sense I think this This is one of the most important inputs to human progress. One of the like retirement things I fantasize about doing is writing a book of everything I've learned about how to build an organization in a culture that does this thing, not the organization that just copies what everybody else has done. Because I think this is something that the world could have a lot more of. It's limited by human talent, but there's a huge amount of wasted human talent because Because this is not an organization, style, a culture, whatever you want to call it, that we are all good at building.

18:01So I'd love way more of that, and that is, I think, the thing most special about us. Sam, how is human talent wasted? Oh, there's just a lot of really talented people in the world that are not working to their full potential, because they work at a bad company or they live in a country that doesn't support any good companies or, along the still other things. I mean, the one of the things I'm most excited about with AI is I hope it'll get us much better than we are now at helping get everyone to their max potential, which we are nowhere, nowhere near. There's a lot of people in the world that I'm sure would be phenomenal AI researchers had their life paths just going a little bit differently.

18:39Sam, you've had an incredible journey, again, sorry for the off script, you've had an incredible journey over the last years through unbelievable hyper growth. You say about writing a book there in retirement.

18:54If you changed your leadership most significantly. Well, I think the thing that has been most unusual for me about these last couple of years is just the rate at which things have changed at a normal company. You get time to go from zero to 100 million in revenue, 100 million to a billion, billions to 10 billion. You don't have to do that in like a two year period. And you don't have to like build the company. We had a research that, but we really didn't have a company in the sense of a traditional Silicon Valley startup that's scaling and serving lots of customers or whatever, having to do that so quickly, there's just like a lot of stuff that I was supposed to get more time to learn than I got.

19:32What did you not know that you would have liked more time to learn? I mean, I would say like, what did I know? One of the things that just came to mind out of like a rolling list of 100 is how hard it is, how much active work it takes to get the company to focus not on how you grow the next 10 % but the next 10x. And growing the next 10 % it's the same things that worked before, we'll work again. But to go from a company doing say like a billion to ten billion dollars in revenue requires a whole lot of change and it is not the sort of like let's do last week what we did this week mindset. And in a world where people don't get time to even get caught up on the basics because growth is just so rapid, I badly underappreciated the amount of work it took to be able to like keep charging at the next big stuff forward while still not neglecting everything else that we have to do.

20:29There's a big piece of internal communication around that and how you sort of share information, how you build the structures to like get the company to get good at thinking about 10X more stuff or bigger stuff or more complex stuff every eight months, 12 months, whatever. There's a big piece there about planning about how you balance what has to happen today and next month with the long lead pieces you need in place for to be able to execute in a year or two years with, you know, build out of a compute or even things that are more normal, like planning ahead enough for like office space in a city like San Francisco is surprisingly hard at this kind of rate.

21:06So there was either no playbook for this or someone had a secret playbook they didn't give me. We've all just sort of fumbled our way through this, but there's been a lot to learn on the fly. God, I don't know if I'm going to get into trouble for this, but I'll ask it anyway. And if so, I'll deal with it later. Keith Reboy, did a talk and he said about, you should hire incredibly young people under 30. And that is what Peter Teal taught him. And that is the secret to building great companies. I'm intrigued when you think about this book that you write in retirement. And that advice, you build great companies by building incredibly young, hungry, ambitious people who are under 30.

21:42And that is the mechanism. And I think I was 30 when we started opening our eye, or at least they're about, so I wasn't that young. It seemed to work okay so far. The question is how do you think about hiring incredibly young under 30s as this Trojan horse of youth energy ambition, but less experience, or the much more experienced, I know how to do this, I've done it before. I mean, the obvious answer is you can succeed with hiring both classes of people. Like I was just like right before this, I was sending someone a Slack message about there was a guy that we recently heard on one of the teams I don't know how old he is but low 20s probably doing just insanely amazing work And I was like can we find a lot more people like this?

22:22This is just like off the charts building. I don't get how these people can be so good so young But it clearly happens and when you can find those people they bring amazing fresh perspective energy Whatever else on the other hand when you're like designing some of the most complex and massively expensive computer systems that humanity has ever built, actually like piece of infrastructure of any sort, then I would not be comfortable taking a bet on someone who is just sort of like starting out where the stakes are higher. So you want both, and I think what you really want is just like an extremely high talent bar of people at any age.

22:57And a strategy that said I'm only gonna hire younger people or I'm only gonna hire older people, I believe would be misguided. It's not quite the framing that resonates with me, but the part of it that does and One of the things that I feel most grateful about Y -combinator 4 is inexperienced does not inherently mean not valuable and there are Incredibly high potential people at the very beginning of their career that can create huge amounts of value We as a society should bet on those people and it's a great thing. I am gonna return to some and the relevance of the schedule is I'm really going to get told off.

23:32But anthropics models have been sometimes cited as being better for coding toss. Why is that? Do you think that's fair? And how should developers think about when to pick OpenAI versus a different provider? Yeah, they have a model that is greater coding for sure, and it's impressive work. I think developers use multiple models most of the time. And I'm not sure that's all going to evolve as we head towards this more a gentrified world, but I sort of think there's just going to be a lot of AI everywhere and something about the way that we currently talk about it or think about it feels wrong. Maybe if I had to describe it, we will shift from talking about models to talking about systems, but that'll take a while.

24:15When we think about scaling models, how many more model iterations do you think scaling laws will hold true for? It was a common refrain that it won't last for long and it seems to be proving to last longer than people think. Without going into detail about how it's going to happen, the core of the question that you're getting at is the trajectory of model capability, improvement going to keep going like it has been going, and the answer that I believe is yes for a long time. Have you ever doubted that? Totally. Why? Well, we've had like behavior we don't understand, we've had failed training runs, we've all sorts of things, we've had to figure out new paradigms when we kind of get towards the end of one and have to figure out the next.

24:57What was the hardest one to navigate? Well, when we started working on GPT -4, there were some issues that caused us a lot of consternation that we really didn't know how to solve. We figured it out, but there was definitely a time period where we just didn't know how we were going to do that model. And then in this shift to a one and the idea of reasoning models, that was something we had been excited about for a long time. But it was like a long and winding road of research to get here. Is it difficult to maintain morale when it is long and winding roads when training runs can fail. How do you maintain morale in those times?