How To Build The Future: Sam Altman

0:00We said from the very beginning we were going to go after AGI at a time when in the field you weren't allowed to say that because that just seemed impossibly crazy. I remember a rash of criticism for you guys at that moment. We really wanted to push on that. And we were far less resourced than DeepMind and others. And so we said, OK, they're going to try a lot of things. And we've just got to pick one and really concentrate. And that's how we can win here. Most of the world still does not understand the value of a fairly extreme level of conviction on one bet. That's why I'm so excited for startups right now.

0:32It is because the world is still sleeping on all of this to such an astonishing degree.

0:43We have a real treat for you today. Sam Altman, thanks for joining us. Thanks, Gary. This is actually a reboot of your series, How to Build the Future. And so welcome back to the series that you started. That was like eight years ago. I was trying to think about that, something like that. That's wild. I'm glad it's being rebooted. That's right. Let's talk about your newest essay on the age of intelligence. You know, is this the best time ever to be starting a technology company? Let's at least say it's the best time yet. Hopefully there'll be even better times in the future. I sort of think with each successive major technological revolution, you've been able to do more than you could before.

1:20And I would expect the companies to be more amazing and impactful and everything else. So, yeah, I think it's the best time yet. Big companies have the edge when things are like moving slowly and not that dynamic. and then when something like this or mobile or the internet or semiconductor revolution happens or probably like back in the days of the industrial revolution that was when upstarts had their have their edge so yeah this is like and it's been a while since we've had one of these so this is like pretty exciting in the essay you actually say a really big thing which is asi super intelligence is actually thousands of days away maybe i mean that's our hope i guess whatever But that's a very wild statement.

2:04Yeah. Tell us about it. I mean, that's big. That is really big. I can see a path where the work we are doing just keeps compounding. And the rate of progress we've made over the last three years continues for the next three or six or nine or whatever. You know, nine years would be like 3 ,500 days or whatever. if we can keep this rate of improvement or even increase it, like that system will be quite capable of doing a lot of things. I think already even a system like O1 is capable of doing like quite a lot of things. From just like a raw cognitive IQ on a closed end, well-defined task in a certain area, I'm like, hmm, O1 is like a very smart thing.

2:48And I think we're nowhere near the limit of progress. I mean, that was an architecture shift that sort of unlocked a lot. And what I'm sort of hearing is that these things are going to compound. We could hit some like unexpected wall or we could be missing something. But it looks to us like there's a lot of compounding in front of us still to happen. I mean, this essay is probably the most techno-optimist of almost anything I've seen out there. Some of the things we get to look forward to, fixing the climate, establishing a space colony, the discovery of all of physics, near limitless intelligence and abundant energy.

3:23I do think all of those things and probably a lot more we can't even imagine are maybe not that far away. And one of, and I think it's like tremendously exciting that we can talk about this even semi-seriously now. One of the things that I always have loved most about YC is it encourages slightly implausible degrees of technical optimism and just a belief that like, ah, you can figure this out. And, you know, in a world that I think is like sort of consistently telling people, this is not going to work. You can't do this thing. You can't do that. I think the kind of early PG spirit of just encouraging founders to like think a little bit bigger is like, it is a special thing in the world.

4:03The abundant energy thing seems like a pretty big deal. You know, there's sort of path A and path B, you know, if we do achieve abundant energy, it seems like this is a real unlock. Almost any work, not just, you know, knowledge work, but actually like real physical work could be unlocked with robotics and with language and intelligence on tap. Like there's a real age of abundance. I think these are like the key two in the two key inputs to everything else that we want there's a lot of other stuff of course that matters but the unlock that would happen if we could just get truly abundant intelligence truly abundant energy what we'd be able to make happen in the world like both like come up with better ideas more quickly and then also like make them happen in in the physical world like to say nothing of it'd be nice to be able to run lots of AI and that takes energy too.

5:01I think that would be a huge unlock and the fact that it's, I'm not sure to be whether, like whether to be surprised that it's all happening at the same time or if this is just like the natural effect of an increasing rate of technological progress, but it's certainly a very exciting time to be alive and great time to do a startup. Well, so we sort of walked through this age of abundance, you know, maybe robots can actually manufacture, do anything, almost all physical labor can then result in material progress, not just for the most wealthy, but for everyone. You know, what happens if we don't unleash unlimited energy?

5:36If, you know, there's some physical law that prevents us from exactly that. Solar plus storage is on a good enough trajectory that even if we don't get a big nuclear breakthrough, we would be like, okay-ish. But for sure, it seems that driving the cost of energy down, the abundance of it up has like a very direct impact on quality of life. And eventually we'll solve every problem in physics. So we're going to figure this out. It's just a question of when. And we deserve it. There's, you know, someday we'll be talking not about fusion or whatever, but about the Dyson sphere. And that'll be awesome too.

6:17Yeah. This is a point in time, whatever feels like abundant energy to us will feel like not nearly enough to our great grandchildren. And there's a big universe out there with a lot of matter. Yeah. I wanted to switch gears a little bit to sort of your earlier you were mentioning Paul Graham, who brought us all together, really created Y Combinator. He likes to tell the story of how, you know, how you got into YC was actually you were a Stanford freshman. And he said, you know what, this is the very first YC batch in 2005. And he said, you know what, you're a freshman, and YC will still be here next time.

6:54You should just wait. and you said, I'm a sophomore and I'm coming. And you're widely known in our community as one of the most formidable people. Where do you think that came from? That one story, I think I would be happy if that drifted off. Well, now it's purely immortalized here. Yeah, here it is. My memory of that is that I needed to reschedule an interview one day or something and PG tried to say, like, I'll just do it next year or whatever. And then I think I said some nicer version of I'm a sophomore and I'm coming, but yeah, you know, these things get slightly apocryphal. It's funny. I don't, and I say this with no false modesty.

7:38I don't like identify as a formidable person at all. In fact, I think there's a lot of ways in which I'm really not. I do have a little bit of a, just like, I don't see why things have to be the way they are. And so I'm just going to like do this thing that from first principles seems like fine. And I always felt a little bit weird about that. And then I remember one of the things I thought was so great about YC and still that I care so much about YC about is it was like a collection of the weird people who are just like, I'm just going to do my thing. The part of this that does resonate as a like accurate self-identity thing is I do think you can just do stuff or try stuff a surprising amount of the time and i think more of that is a good thing and then i think one of the things that both of us found at yc was a bunch of people who all believed that you could just do stuff for a long time when i was trying to like figure out what made yc so special i thought that it was like okay you have this like very amazing person telling you i you can do stuff i believe in you.

8:48And as a young founder, that felt so special and inspiring. And of course it is. But the thing that I didn't understand until much later was it was the peer group of other people doing that. And one of the biggest pieces of advice I would give to young people now is finding that peer group as early as you can was so important to me. And I didn't realize it was something that mattered. I kind of thought, ah, like I have, you know, I'll figure it out on my own. But man, being around like inspiring peers, so, so valuable. What's funny is both of us did spend time at Stanford. I actually did graduate, which is, I probably shouldn't have done that, but I did.

9:28Stanford's great. You pursued the path of a far greater return by dropping out. But that was a community that purportedly had a lot of these characteristics, but I was still beyond surprised at how much more potent it was with a room full of founders. It was, I was just going to say the same thing. Actually, I liked Sanford a lot. Yeah. But I was, I did not feel surrounded by people that made me like want to be better and more ambitious and whatever else. And to the degree I did, the thing you were competing with your peers on was like, who was going to get the internship at which investment bank, which I'm embarrassed to say, I fell on that trap.

10:07This is like how powerful peer groups are. it's a very easy decision to not go back to school after like seeing what the like YC vibe was like yeah there's a powerful quote by Carl Jung that I really love it's you know the world will come and ask you who you are and if you don't know it will tell you it sounds like being very intentional about who you want to be and who you want to be around as early as possible is very important. Yeah, this was definitely one of my takeaways, at least for myself, is no one is immune to peer pressure. And so all you can do is like pick good peers. Yeah. Obviously, you know, you went on to create Looped, you know, sell that, go to Green Dot, and then we ended up getting to work together at YC.

10:52Talk to me about like the early days of YC research. Like one of the really cool things that you brought to YC was this experimentation. And you sort of, I mean, I I remember you coming back to partner rooms and talking about some of the rooms that you were getting to sit in with like the Larry and Sergei's of the world and that, you know, AI was some sort of at the tip of everyone's tongue because it felt so close. And yet it was, you know, that was 10 years ago. The thing I always thought would be the coolest retirement job was to get to like run a research lab. And it was not specific to AI at that time.

11:31Well, we started talking about YC research. Well, not only was it going to, it did end up funding like a bunch of different efforts. And I wish I could tell the story of like, if it was obvious that AI was going to work and be the thing. But we tried a lot of bad things too. Around that time, I read a few books on like the history of Xerox, Park and Bell Labs and stuff. And I think there were a lot of people, like it was in the era of Silicon Valley at the time, that we need to like have good research labs again. and I just thought it would be so cool to do. And it was sort of similar to what YC does in that you're going to allocate capital to smart people and sometimes it's going to work and sometimes it's not going to.

12:11And I just wanted to try it. AI for sure was having a mini moment. This was like kind of late 2014, 2015, early 2016 was like this super intelligence discussion, like the book Super Intelligence was happening. Boestrom, yeah. Yeah. The DeepMind had a few impressive results, but a little bit of a different direction. I had been an AI nerd forever, so I was like, oh, it'd be so cool to try to do something, but it's very hard to say. Was ImageNet out yet? ImageNet was out. Yeah. Yeah. For a while at that point. So you could tell if it was a hot dog or not. You could sometimes. Yeah. That was getting there.

12:48Yeah. How did you identify the initial people you wanted involved in YC research and OpenAI? I mean, Greg Brockman was early. In retrospect, it feels like this movie montage. And there were like all of these, like, you know, at the beginning of like the Bank Eyes movie when you're like driving around to find the people and whatever. And they're like, you son of a bitch, I'm in. Right. Like Ilya, I like heard he was really smart. And then I watched some video of his and he's also, he's extremely smart, like true, true, genuine genius and visionary. But also he has this incredible presence. And so I watched this video of his on YouTube or something.

13:25I was like, I got to meet that guy. And I emailed him. He didn't respond. So I just like went to some conference he was speaking at. And we met up. And then after that, we started talking a bunch. And then like Greg, I had known a little bit from the early Stripe days. What was that conversation like, though? It's like, I really like what your idea is about AI. And I want to start a lab. Yes. And one of the things that worked really well in retrospect was we said from the very beginning, we were going to go after AGI at a time when in the field you weren't allowed to say that because that just seemed impossibly crazy and, you know, borderline irresponsible to talk about.

14:03So that got his attention immediately. It got all of the good young people's attention and the derision, derision, whatever that word is, of the mediocre old people. And I felt like somehow that was like a really good sign and really powerful. And we were like this ragtag group of people. I mean, I was the oldest by a decent amount. I was like, I guess I was 30 then. And so you had like these people who were like, those are these irresponsible young kids who don't know anything about anything. And they're like saying these ridiculous things. And the people who that was really appealing to, I guess are the same kind of people who would have said like, it's a, you know, I'm a sophomore and I'm coming or whatever.

14:39And they were like, let's just do this thing. Let's take a run at it. And so we kind of went around and met people one by one and then in different configurations of groups and it kind of came together over the course of in fits and starts but over the course of like nine months and then it started i mean and then it started started happening and one of my favorite like memories of all of open ai was ilia had some reason that google or something that we couldn't start in we announced in december of 2015 but we couldn't start until january of 2016 so like January 3rd something like that of 2016 or like very early in the month people come back from the holidays and we go to Greg's apartment maybe there's 10 of us something like that and we sit around and it felt like we had done this monumental thing to get it started and everyone's like so what do we do now and what a great moment it reminded me of when startup founders work really hard to like raise a round and they think like oh I accomplished this great thing we did it we did it and then you sit down and say like fuck now we gotta like figure out what we're gonna do it's not time for popping champagne that was actually the starting gun and now we gotta run yeah and you have no idea how hard the race is going to be it took us a long time to figure out what we're gonna do um but one of the things that i'm really amazingly impressed by ilia in particular but really all of the early people about is although it took a lot of twists and turns to get here, the big picture of the original ideas was just so incredibly right.

16:14And so they were like up on like one of those flip charts or whiteboards, I don't remember which, in Greg's apartment. And then we went off and, you know, did some other things that worked or didn't work or whatever. And some of them did. And eventually now we have this like system. and it feels very crazy and very improbable looking backwards that we went from there to here with so many detours on the way but got where we were pointing. Was deep learning even on that flip chart initially? Yeah. I mean, more specifically than that, like do a big unsupervised model and then solve RL was on that flip chart.

16:50One of the flip charts from a very, this is before Greg's apartment, but from a very early offsite. I think this is right. I believe there were three goals for the effort at the time. It was like, figure out how to do unsupervised learning, solve RL, and never get more than 120 people. Missed on the third one. But the predictive direction of the first two is pretty good. So deep learning, then the second big one sounded like scaling. Like the idea that you could scale. That was another heretical idea that people actually found even offensive. I remember a rash of criticism for you guys at that moment.

17:31When we started, yeah, the core beliefs were deep learning works and it gets better with scale. And I think those were both somewhat heretical beliefs. At the time, we didn't know how predictably better it got with scale. That didn't come for a few years later. It was a hunch first and then you got the data to show how predictable it was. But people already knew that if you made these neural networks bigger, they got better. Like that was, we were sure of that before we started. And what took the like, word that keeps coming to mind is like religious level of belief was that that wasn't going to stop.

18:09Everybody had some reason of, oh, it's not really learning. It's not really reasoning. It can't really do this. It's, you know, it's like a parlor trick. And these were like the eminent leaders of the field. And more than just saying you're wrong, they were like, you're wrong. And this is like a bad thing to believe or a bad thing to say. It was that there's got it. You know, this is like, you're going to perpetuate an AI winter. You're going to do this. You're going to do that. And we were just like looking at these results and saying they keep getting better. Then we got the scaling results.

18:41It just kind of breaks my intuition even now. and at some point you have to just look at the scaling laws and say we're going to keep doing this and this is what we think it'll do and it also it was starting to feel at that time like something about deep learning was just this emergent phenomenon that was really important and even if we didn't understand all of the details in practice yet which obviously we didn't and still don't, that there was something really fundamental going on. It was the PG-ism for this is we had discovered a new square in the periodic table. And so we really wanted to push on that.

19:22And we were far less resourced than DeepMind and others. And so we said, okay, they're going to try a lot of things. And we've just got to pick one and really concentrate. And that's how we can win here, which is totally the right startup takeaway. And so we said, well, we don't know what we don't know. we do know this one thing works. So we're going to really concentrate on that. And I think some of the other efforts were trying to outsmart themselves in too many ways. And we just said, we'll just, we'll do the thing in front of us and keep pushing on it. Scale is this thing that I've always been interested in at kind of just the emergent properties of scale for everything, for startups, turns out for deep learning models, for a lot of other things.

20:03I think it's a very underappreciated property and thing to go after. And I think it's, you know, When in doubt, if you have something that seems like it's getting better with scale, I think you should scale it up. I think people want things to be, you know, less is more, but actually... More is more. More is more. We believed in that. We wanted to push on it. I think one thing that is not maybe that well understood about OpenAI is we had just this... Even when we were, like, pretty unknown, we had a crazy talented team of researchers. You know, if you have, like, the smartest people in the world, you can push on something really hard.

20:37Yeah, and they're motivated. And or you created sort of one of the sole places in the world where they could do that. Like one of the stories I heard is just even getting access to compute resources. Even today is this crazy thing. And embedded in some of the criticism from maybe the elders of the industry at the moment was sort of that, you know, you're going to waste a lot of resources. And somehow that's going to result in an AI winter. People won't give resources anymore. It's funny. People were never sure if we were going to waste resources or if we were doing something kind of vaguely immoral by putting in too much resources.

21:15And you were supposed to spread it across lots of bets rather than like conviction on one. Most of the world still does not understand the value of like a fairly extreme level of conviction on one bet. And so we said, OK, we have this evidence. We believe in this thing. We're going to at a time when like the normal thing was we're going to spread against this bet and that bet and that bet. You're a definite optimist. You're a definite optimist. And I think across like many of the successful YC startups, you see a version of that again and again. Yeah, that sounds right. When the world gives you sort of pushback and the pushback doesn't make sense to you, you should do it anyway.

21:48Totally. One of the many things that I'm very grateful about getting exposure to from the world of startups is how many times you see that again and again and again. And before, I think before YC, I really had this deep belief that somewhere in the world, there were adults in charge, adults in the room, and they knew what was going on. And someone had all the answers. And, you know, if someone was pushing back on you, they probably knew what was going on. And the degree to which I now understand that, you know, to pick up the earlier phrase, you can just do stuff. You can just try stuff. No one has all the answers.

22:24There are no like adults in the room that are going to magically tell you exactly what to do. And you just kind of have to like iterate quickly and find your way. That was like a big unlock in life for me to understand. There is a difference between being a high conviction just for the sake of it. And if you're wrong and you don't adapt and you don't try to be like truth seeking, it still is really not that effective. The thing that we tried to do was really just believe whatever the results told us and really kind of try to go do the thing in front of us. And there were a lot of things that we were high conviction and wrong on.

23:03But as soon as we realized we were wrong, we tried to like fully embrace it. Conviction is great until the moment you have data one way or the other. And there are a lot of people who hold on it past the moment of data. So it's iterative. It's not just they're wrong and I'm right. You have to go show your work. But there is a long moment where you have to be willing to operate without data. And at that point, you do have to just sort of run on conviction. Yeah. It sounds like there's a focusing aspect there, too. Like, you had to make a choice, and that choice had better... You know, you didn't have infinite choices.

23:36And so, you know, the prioritization itself was an exercise that made it much more likely for you to succeed. I wish I could go tell you, like, oh, we knew exactly what was going to happen. And it was, you know, we had this idea for language models from the beginning. And, you know, we kind of went right to this. But obviously, the story of OpenAI is that we did a lot of things that helped us develop some scientific understanding, but were not on the short path. If we knew then what we know now, we could have speedrun this whole thing to like an incredible degree. It doesn't work that way. Like you don't get to be right at every guess.

24:10And so we started off with a lot of assumptions, both about the direction of technology, but also what kind of company we were going to be and how we were going to be structured and how AGI was going to go and all of these things. and we have been like humbled and badly wrong many many many times and one of our strengths is the ability to get punched in the face and get back up and keep going this happens for scientific bets for uh you know being willing to be wrong about a bunch of other things we thought about how the world was going to work and what the sort of shape of the product was going to be again we had no idea or i at least had no idea maybe alec radford did i had no idea that language models were going to be the thing.

24:54You know, we started working on robots and agents playing video games and all these other things. Then a few years later, GPT-3 happened. That was not so obvious at the time. It sounded like there was a key insight around positive or negative sentiment around NGP-1. Even before GPT-1. Oh, even before. I think the paper was called The Unsupervised Sentiment Neuron. And I think Alec did it alone. By the way, Alec is this unbelievable outlier of a human. And so he did this incredible work, which was just looking at, he noticed there was one neuron that was flipping positive or negative sentiment as it was doing these generative Amazon reviews, I think.

25:37Other researchers might have hyped it up more or made a bigger deal out of it or whatever, but it was Alex, so it took people a while to, I think, fully internalize what a big deal it was. And he then did GPT-1 and somebody else scaled it up into GPT-2. But it was off of this insight that there was something amazing happening where, at the time, unsupervised learning was just not really working. So he noticed this one really interesting property, which is there was a neuron that was flipping positive or negative with sentiment. And yeah, that led to the GPT series. I guess one of the things that Jake Heller from Case Text, I think of him as maybe, I mean, not surprisingly, a YC alum who got access to both 3, 3.5, and 4.

26:27And he described getting 4 as sort of the big moment revelation. Because 3.5 would still do, I mean, it would hallucinate more than he could use in a legal setting. and then with 4 it reached the point where if he chopped the prompts down small enough into workflow he could get it to do exactly what he wanted and he built you know huge test cases around it and then sold that company for 650 million dollars so it's uh you know i think of him as like one of the first to commercialize gpd4 in a relatively grand fashion i remember that conversation with him with one GPT-4. Like that was one of the few moments in that thing where I was like, okay, we have something really great on our hands.