Ep 66: Bob McGrew — the Superstar Palantir Alum Leading OpenAI's Transformative Research Projects

0:00In two weeks we went from not being able to grab a ball with a claw to having a five finger humanoid robot hand solving Rubik's cube in simulation. And I was like, wow, wow. That's a big leap. And so to me, that was what convinced me that this was actually possible, that we actually had found a mechanism that if you could figure out how to pour enough compute and pour enough data into a neural network, there's really no limit to how good it could get.

0:35Bob McGrew is an old friend from Stanford. We were in a couple groups there together. He was ahead of me in computer science. He joined PayPal a year before me. You know, I learned a lot from him over the years. We convinced him to come on early at Palantir. He was a key early leader, helped run engineering, helped run product, really helped make Palantir into a great company. Normally, that'd be enough for a good reason to talk to Bob. But you know what? He joined OpenAI first in 2016, full-time in 2017. This was before the Transformer paper. I have no idea how he knew this was the place to go.

1:04OpenAI has transformed the world. He has the front row seat there, helping run research, helping really run a lot of the efforts going on at OpenAI. This is the guy to talk to, to hear what's happening in the AI world. I'm looking forward to having you meet him. I'm Joe Lonsdale. Welcome to the American Optimist. Bob McGrew, thanks for joining us today. Great to see you again. Bob is an old friend. We were in a couple of groups at Stanford together, including FI-Sci. And you were at PayPal a year before me. 2001, 2002. What were you doing at PayPal? I was working on cryptography. And tell us a little bit about your background.

1:33How did you end up at Stanford, at PayPal? Where are you from, Bob? Well, originally I came from a small town in Oklahoma. And my parents were college professors. And my dad was a computer science professor. And I remember, I think third grade, he bought me a book where two kids use computer programs to solve problems for detectives. And I was hooked. And so my whole childhood, I would buy these books and I would always be writing programs. But again, small town, Oklahoma. and this was during the first web boom. And so I always knew I wanted to go to Stanford. I never actually thought I'd get in, but I did.

2:12That's pretty funny, actually. I didn't realize that you're solving problems for detectives in the book. And then you go to Palantir, which also kind of solved problems to help catch the bad guys. It's a crazy coincidence, but I think it just shows that I knew what I wanted to do from a very young age. So you joined OpenAI what year? 2017. 2017. So for the last six years, you've been helping run things at OpenAI. But let's talk about Palantir a little bit. You know, I interviewed Alex Karp earlier, our good old Dr. Karp. And he talked a lot about how it took a really special type of genius to create what we did in terms of the engineers at, you know, Pounder Gotham.

2:44What was Pounder Gotham? What were you working on? Why was it hard? Well, I think the really funny thing about Gotham was the hardest part was we had no idea what it was supposed to be at the beginning. So the goal was to build this platform for the intelligence community. So basically software for spies. But as you remember, the first hard problem about a spy is actually finding one. And then once you find one, they don't actually want to tell you what they're doing. They want to tell you cool stories about using guns and little devices and stuff. But yeah, not really about the information architecture.

3:15Or you try to ask them what they're doing. They're like, oh, that's classified. And they give you some sort of tortured baseball analogy. And so instead of all the UX books, you interview your customers, you find out what their problems are, and you build them to solve their problems. And we couldn't do that. And so, basically, what we had to do was we had to build the product we thought they needed, which was wrong, and show it to them. And they would tell us we were wrong. They'd be like, oh, this is crap. Why don't you build this other thing instead? And we would do that. And you and Stefan would go into DC and show this to them and bring back that feedback.

3:46It was a combination, Rick. Because I do think there were certain first principles, but we all kind of could figure out they would definitely need. And then there was the iteration of what they actually needed. And so, we had to have a lot of both sides. We knew it was something about data, that somehow we had to get their data in, and that we would let them ask questions about their data. But we were totally wrong at the beginning about what kind of data they had. They had a lot less structured data than we assumed. Exactly. Yeah. It turns out, intelligence community, it's all about reading documents and trying to understand and parse documents and figure out who the people are in the documents.

4:13And if the person in this document is the same as that description they have in the other document. They spent over$30 billion on databases and gathering data, so I assumed that they had more structured data. So that was pretty complicated. Yeah. Yeah. And that was, that was mostly wrong. I think they had it and they didn't know what to do with it. Do you remember any of the characters? One of my favorites was Sir Mark. Do you remember him? I do. Yeah. He was the one who said that if you don't get your gun unless you're going to use it, and if you're going to use it to pull the trigger twice, he's the MI6 guy.

4:39You have any other favorites? I remember, I think I have to stick to first names here, but I remember our first customer at the agency. Her name was Sarah. And she was the only person that believed in us. I think the really crazy thing about Palantir was how long it took to get a product that actually worked. I think it took three years before we had anyone using it to solve. I remember a few people almost quit right around year three, right before that happened. And it was like this thing where like people were so exhausted, like this is just crazy and it's not going to work. And then they kind of, we got to stay on and then we kind of got over the hump and people started using it.

5:13I remember that. I remember, you know, and, and, and this, this is one of those things about the really great engineers is that a really great engineer wants certainty. Right. They don't want to, you know, they've been doing something for three years. All they can see are all the problems that are in it. And, you know, I remember sitting down and having a conversation with our best engineer and just sort of convincing him to stick it out for nine more months. And he built the security system that we ended up working with. It was a critical piece to allowing us to actually deploy. And then he quit.

5:43Yeah. But if he had. Well, thank you for holding the team together. We were both fighting hard to hold it together because it was kind of like, what are these kids doing at that point? I don't know if you know, you joining was really, really critical to Peter giving us our money for the second time. Because he was like, you guys got Bob to join? How did you do that? I did not know that. That was actually, yeah. So thank you. You're a good indicator for Peter Thiel. You've kept that a secret from me for all these years. Wow. I forgot to tell you earlier on. We all did very well from that. But you may even be doing even better financially from OpenAI.

6:14Not that we're talking about finances, but this is amazing. You guys have just crushed it in terms of changing the world the last six, seven years. How did you find OpenAI? What's the story there? Well, when I left Palantir, my PhD before I started Palantir was in AI. And I left because I felt AI wasn't happening. And this was 2005. And in fact, AI was not happening in 2005. Yes, it would have been very hard to work on AI 20 years ago. The key thing about AI was actually just big data. And you didn't need AI to unlock it. The last thing was that everyone who went to work on this, it was just a bad choice forever.

6:45It was exciting, but then it was never the right choice. Yeah, until it was. Until it was. So how'd you know that it was? How'd you have the intuition that it might be? Well, so what happened is, in, I think, 2011, there was a paper that was published by Ilya Sutskiver and some other people who, Ilya Sutskiver, chief scientists at OpenAI, where they basically reinvented neural networks. And they showed that if you ran them on GPUs, you could plug in huge amounts, much bigger amounts of compute and much bigger amounts of data. And neural networks went from being, oh, that one thing that you can kind of use for handwriting to something that was actually the best way to identify images.

7:18And for some of our listeners who aren't just technical, we're talking about a neural network. That's something that gives iterative feedback on things. How would you explain neural networks? So, a neural network, the analogy that most people use is that it's like the brain. And it is, but at a very, very high level. So, a neuron is basically, you can think of it as having connections to other neurons. And the strength of those connections determines when the first neuron, the lower level neuron, fires. Then that makes the higher level neuron fire. And if you train these, you do it by showing them the right answer and then basically doing what's called backpropagating the error from the top of the network where the answer is all the way back down through the whole network.

8:01Is it a power detection type of situation then? Or what is it? Yeah, you're building hierarchical representation. So at the very bottom, think about a vision network because it's easy to visualize. So at the very bottom, the neurons are detecting edges. And then they go up a little bit higher and they're detecting corners. And if you go high enough, they're detecting wheels. And then above that, they're detecting cars. If you have quarters in this way, then it's a square. This could be a wheel, and then it could be a car. Yeah, and at some point, there's a Joe Lonsdale neuron that is out there.

8:27And individual people, you can actually find these neurons in the networks. I remember reading On Intelligence by Hawkins, who didn't actually end up solving all these problems. But he said there's like six layers of the neocortex that is part of the vision system. Is that right, or is there actually a lot more than that? You know, I'm not a cognitive scientist. So the other thing is, I think the analogy between the brain and the neural networks, it's very inspirational, but you can take it too far. It's very imprecise. What OpenAI did is you basically didn't base it on the brain. You based it on building it up from scratch, from just the first principles.

8:54So I think that the thesis behind OpenAI, when I joined in 2017, was that neural networks were the final architecture that could take AI all the way to human-level intelligence, or AGI. But wasn't there a transformer breakthrough that was really important, though, right around 2017? Yeah, that's right. So, you know, the first couple of years of OpenAI, we were using sort of standard neural networks. And then there was a paper at Google. Some people came up with an idea called Transformers that allow you to take better understanding of the context. So, for example, you could look at documents. And now you could really understand the preceding three or four pages when the prior architectures had struggled just to understand, like, the previous 10 words.

9:38What's the intuition for why Transformers worked better? What's going on there? It's basically because they have a notion of attention. So you can pay attention to particular words in a document versus previous architecture had a notion of memory. So you could sort of remember the words that you had seen. But it's much easier to be able to look at a sheet of paper and sort of see the different words and think about reading those words. Pay attention to what matters. Pay attention to what matters. Which is kind of how the brain works. Things jump out at us, obviously, when we look at things. Yeah, it's very intuitive.

10:11The intuition, I know this is not based, you're not a cognitive scientist, but I want to try to build intuition here for people. My understanding, the big breakthrough we had for how the brain works is you're constantly predicting what you're going to see next. Like, this may be why we see ghosts, but either way, your brain's constantly looking at what it thinks it expects to see. And then if something is not unexpected, it kind of jumps out at you. Is there anything like that with transformers or there's nothing like that where it's trying to predict things? Well, that's exactly how you train a model like GPT-4.

10:36So, you know, if you take it to train a model like GPT-4, basically we take all of the text, everything we can scrape off the Internet. And what the model is trying to do is it goes character by character and it's trying to predict the next character. So, you know, if it sees, you know, the rain in Spain falls mainly on the it's going to guess, oh, that's a P and it's going to be plain. And it does this over and over again for huge amounts of documents, like trillions of characters. And over time, it seems as though that yields something that looks like intelligence. The prediction seems very, very similar to intelligence, which may be what we're doing, too.

11:13Yeah. At this point, I think we understand neural networks a lot better than we understand the brain. So whenever someone talks to me about the brain, I always think, well, what does a neural network do? And then probably, actually, the brain does that. Do you think of yourself as a neural network now? I do, actually. I actually think of my kids as neural networks. How does that change your interactions with them? you know as i was starting at open ai first i worked on robotics and then i worked on we did some problems with math and then programming and early on you know the kids would always get the problems before the robots were and now it's the other way around now the neural networks are better are you know better at solving problems than the kids because it given you any intuition on how to train young minds uh yeah you just show them lots and lots of things you just have them read just Show them three trillion things.

11:58I mean, that's probably how you and I learned, right? You know, just read a lot of books when we were kids. I guess there's a thing about attention, too. I've always found that kids do better when they're confident. I don't know if the computers need to be given confidence, although maybe that's a signal for something else like attention. Well, that's actually one of the cool things that we've discovered with GPT-3 and GPT-4 is that the models are trained on the Internet. And they're reading everything, right? Then they try to act like what they've seen, right? They're predicting. And so they don't know whether to predict being someone who's dumb or someone who's really smart.

12:31That's what you have to tell them they're really smart when you're talking to them. It's so funny. Exactly. Like you're a confident, you know, powerful physicist who knows everything. The best physicist in the world. Now answer my question. And you sound like Shakespeare because it's fun if you make him sound cool. That's right. Or you have to do everything in a limerick. That's pretty good. So let's back up a little bit. So you joined OpenAI in 2017. What's your role in OpenAI? So I'm the VP of research. What does that mean?

12:58So, if you're the VP of engineering at a company, so Palantir, what I spent all my time doing was basically, fundamentally, you're trying to figure out, are the right people working on the right problems? And figuring out what's most important and finding people to work on that. You would get in the problems yourself, too, but then you're more focused on the people. Yeah, I mean, especially early on, I read every commit and I wrote a lot of the back end for Palantir. But, you know, as the company scales, you know, you as a leader end up focusing more and more on the people problems. And, you know, in a normal company, you're often trying to figure out what the people should be working on.

13:31And at a research company, it actually doesn't work very well that way. So there's this researcher, Aditya Ramesh. He built a model called DALI, which allowed you to take a description and build an image out of it. It's very fun. He spent two years working on that. Wow. For most of those two years, no one other than him believed it would work. I mean, I sort of believed it would work. I really hoped it would work. I made sure he had the GPUs he needed. But in order to have that kind of drive to really push it through for two years, this is your dream. Yeah, you have to be obsessed with it, basically.

14:08Yeah. And I remember, you know, Karp always used to talk about this, that the best engineers were artists. And if you want to keep an engineer happy, what you actually have to do is just let him do his art. Just get out of his way. Let him do his thing. And your job when you're building a company is to have all these brilliant people doing their art. And somehow, magically, it turns into a product that solves a real problem. It seems very unusual to have one person do it for two years. Usually, there's a small team or something. And they're giving feedback and iterating. Yeah. And that's really what's different about research.

14:37There's a lot of things we do that have teams. And I think that's one of the things that we brought to research at OpenAI, is to have teams that can work on problems together. But a lot of the very early stuff is just one person. Wow. Like, this has to be their hill to die on, or it won't happen. So, you have people right now on your team going for a year or two on things that hopefully will work, but it's not clear. That's still a culture there for some of these things. Yeah, yeah. That's really cool. And as we've gotten bigger, I think we've sort of shifted. As a fraction of the company, there's fewer people doing these sort of hills they have to die on that are really early.

15:08And more, we've taken a startup approach to doing research. So, we have teams of people. We have broken out problems. But again, the difference between research and engineering is that if you're trying to tell someone, hey, can you build this user interface? There's not usually a question about whether you can. No one's like, oh, I can't build that. The engineering team is just like, we're going to build this interface. Whereas the research team is, we're going to explore and see if we could do this thing. What were some of those things early on, if you could tell us, from 2017, 18, 19, 20? Other than Dolly, what were some of the crazy research projects?

15:37Yeah, I think the first crazy research project we did was a project called... Basically, we were trying to beat the game Dota 2. So, there's this long history in AI. You beat chess, you beat Go. Dota 2 doesn't have quite the level of class, but it's actually a much harder game. It's a very hard problem. Are you a gamer? Yes, I am a gamer. Well, you know, obviously, I was chess champion, and then the Go thing totally blew me away and freaked me out because that was really scary. And then my favorite part was when they learned to play chess against itself and learn all the openings. Because we always used to have computers, like we'd tell them the openings because they just don't know them.

16:12And then they'd be able to play. And what was really cool was throughout the day, it learned all the old openings. And shifted from playing 15th century players to 19th century players to modern players to then playing even differently. Which is really creepy as a person to watch the computers do hundreds of years of human history in 24 hours. And that's actually one of the really interesting things. You train a model like GPT-3 or GPT-4. It's basically learning what humans have learned by trying to predict what they were doing. The model we trained for Dota 2 or the models you trained for chess or Go is basically doing something different.

16:42It's doing reinforcement learning. It's trying to win. They probably play very differently than people because they're so fast. They can do things we can't do, right? Well, early on, it plays a lot of the same things. And then over time, you see it develop new approaches that humans have never developed before. Yeah, it's because we're not fast enough. Or it also maybe just tries things we haven't tried. Yeah, I think since it's playing itself, it can sort of end up in equilibria that humans would never find. Interesting. And did it end up crushing Dota 2? Yeah, so we ended up beating Dota 2. And I think for me, what was really crazy about this is that when I wanted to open AI, Ilya Suskev, our chief scientist, sat me down and was like, we're going to build AGI.

17:19It's going to be human level intelligence. We're going to do this in 10, 15 years. And I'm like, that would be cool if it were true. But I have no idea how we're going to get there. you know and i would ask him and he'd say well you know we're gonna figure it out and um i remember one week i was sitting there i had spent months you know i was working on robotics not on the doter 2 problem and i was trying to get a robot claw like just a two-finger claw to grab you know a ball it's a hard problem right and i i just couldn't make it work and i remember thinking myself like why are we talking about agi if we can't even have like a claw grab a ball like This is really easy.

17:51We should be able to do this. And at the same time, the Dota 2 team, Jakub Paczoky, who was doing this, had developed a technique that allowed you to put lots of data and compute into solving the problem. And it was really working for Dota. And he came over and started doing this for robotics. And in two weeks, we went from not being able to grab a ball with a claw to having a five-finger humanoid robot hand solving Rubik's Cube in simulation. And I was like, wow. Wow. Wow. That's a big leap. And so to me, that was what convinced me that this was actually possible. That we actually had found a mechanism that if you could figure out how to pour enough compute and pour enough data into a neural network, there's really no limit to how good it could get.

18:37What games are left right now that you can't solve? I think Dota 2 was the last one. Is there really not? I mean, there must be some other more complex human game. So the last really cool game to get solved was Diplomacy. Yeah, I heard of that. Because, you know, what's smart about it is that you have to do a lot of interaction and they have to work with human players, not just with AI. There's a lot of subtle strategy with how you work together versus how you betray people and when you betray people. Very scary. We're teaching them this, but I guess this is like the natural strategy for these things.

19:08It's fascinating. So you can't think of, I mean, they must not be good at baseball, for example, just because the robots aren't good enough. That's true. Yeah. I mean, I think where we are right now is that if you have a job that is, you know, remote, doable remotely, which at this point is most laptop jobs, right? The computers can maybe get better at that. Yeah. It feels really achievable that we're going to be able to get there. This is very unintuitive because forever technology replaced blue collar work. But now technology is replacing the bureaucrats and the blue collar like plumbers are like, we're fine.

19:34But the like annoying lawyer people are like, they're in trouble. I mean, the thing right now is what we're seeing is that technology is basically eliminating the rote parts of the jobs, right? So when I used to program, I remember I would have to write all these XML configs. And nowadays, you just ask GPT-4 and it'll write the XML configs for you. But you want to do something really interesting, then you need to bring your human intelligence to the problem, even if it's just telling the model how to go about it. So you're closer to this than 99.999 % of people in the world. So I want to ask your intuition because I've talked to Sam, obviously, who's involved in this.

20:10And I think you may even be more technical than him and be really close to the research. So, you know, the next 10, 20 years, I mean, are you going to be able to do everything people can do in five or 10 years? Is this like a total change in all of reality by the 2030s? Is there are there actually a lot more steps potentially still like what's going to happen here? Yeah. I mean, the thing about AGI, artificial general intelligence, which is people call AGI, is that it's very clear to me. and I think it's clear to people who are working in the field that progress is going to continue to happen.

20:39What's hard to predict is when you actually are able to do a particular job. When the the there's so many different things that are wrapped up in any particular job that it may turn out that something we didn't even expect was the last thing that that needed to be automated. So I think we're going to have a long time. We may not have like perfectly realistic Westworld robots walking around in like seven or eight years. Robotics? Robotics are probably going to need AI to write that, to figure out how to build robots for us. But I mean, you're going to get... So, that's the other question. I don't know what public or not, so push back on me.

21:15But the thing I've heard from various people is you can probably do GPT 5 and 6 and maybe 7 with people. And then at some point, you're going to need to get to 8 or 9 or whatever. You're going to need GPT itself to do that. If I was running a research group, I'd already be trying to probably figure out how to get this thing to teach itself in different ways. that's probably like there's lots of concepts there. I'd imagine that you're trying to figure out how far are we away from it teaching itself? And is that a thing you guys are working on that you could say? Yeah, I mean, I think that's what everybody's, I think that's sort of the open problem that everybody sees right now is that, you know, the current, as you say, the current GPTs are all just mimicking what humans do.