180 - Andrew Lampinen: Translating Cognitive Psychology to AI

16:02Andrew Lampinen:So I thought it was a really great training environment for thinking about all the different perspectives you could take on issues in cognition and neuroscience.

16:12Elizabeth:I mean, even in academia, like last year's MBCT symposium, this was a big topic like whether is academia going to be competitive against industry and research and I think this was a topic because like last year like a bunch of like Nobel laureates like came from industry not academia and as you said like industry have research teams and they they definitely have a lot more resource especially in like more computational work there's a lot more There's more GPU. I see my neighboring labs or even some of my lab mates who do more computational work, they're fighting for GPU every day. I guess I'm wondering, in this changing landscape of research, what do you think as someone who's experienced both?

17:08Elizabeth:Like, what is the strength of, what is the difference and what is like the strength of academic research?

17:23Andrew Lampinen:A few things. So first of all, like academia is more training focused and I don't think that's a bad thing, right? Like I think for me, I think it was a good path to get trained in academia where you have the space to think about bigger picture issues and you can sort of like, you know, explore different topics. I mean, you can't do that in industry, too. But it's like you have a little bit more freedom during a PhD to like make a bet on a topic for a few years and maybe it doesn't work out and explore some things and meet people with different perspectives and so on. I think that like, you know, the other thing maybe related to what I was saying earlier is that like academia is like at its best, it always has the potential for doing really blue sky exploratory research.

18:18Andrew Lampinen:Right. And that's something that I think in industry, like really depends upon the moment. And there's been like, you know, many cases like this is something like Bell Labs, for example, where there's like or Xerox Park where there's like industry funded research groups that do really basic research that is fundamentally impactful. but like eventually they kind of like burn out one way or so i guess like academia maybe like by not being subject to the same economic pressures as an actual company has more potential perhaps to do more exploratory research and to some extent i think that the resource constraints are a good thing in that sense right because it kind of forces academics to make different bets than industry because, you know, whatever the capex of these companies is now, it's like spending, you know, tens or hundreds of billions of dollars on compute.

19:17Andrew Lampinen:Academia is not going to match that, but like that means that there has to be a different bet taken. And I think that's always good for diversifying the set of things the field is working on as a whole. But yeah, part of the reason that I'm still in industry and I'm happy that I came to industry is that I think there are certain kinds of research areas that academia will have a very hard time keeping up with industry in. And I happen to, at least at the moment, be still excited about, you know, following the research in those areas very actively and participating in it. And I don't think that academia should necessarily try to compete with industry in the area where industry is throwing a lot of money at certain bets.

20:03Andrew Lampinen:that's going to be a hard fight to win but uh it i think academia should lean into its own

20:11Elizabeth:strengths yeah that that's a great perspective so so you went to industry i'm curious what what was so from what we've already talked about it sounds maybe industry academia is doing somewhat of a different texture of research. I'm wondering, like, once you got to industry, like, what was it like? So you graduated from Stanford, got PhD, and then what happened once you went out there?

20:44Andrew Lampinen:Yeah, I feel like the research and industry has changed a lot over the time I've been there. So when I got there, it was, like, quite exploratory blue-sky-y research, as I said earlier. And, you know, for example, I worked on some stuff about almost like philosophy of like, what are symbols and how could a system understand symbols? Very like cognitively questions. And also some more like practical work on like, you know, how do you get good memory systems for agents so that they can learn something and then remember it for a long time. But like, because language models were becoming more and more salient, I had done some work in sort of like topics adjacent to language modeling, like how do you do fast learning of word embeddings and things like that in grad school.

21:36Andrew Lampinen:And so like, I was pretty quickly interested in working on, you know, what are these systems doing? Why are they doing it? And so because of that, I started to shift a bit into working on language models, although I've still kept some of the more embodied agent research that I did.

21:54Elizabeth:Sorry. So in grad school, I think you said you worked on how people are able to learn fast, like basically how people are able to do one-shot learning. Yeah.

22:05Andrew Lampinen:So I guess the overall theme of my grad school research was kind of like, how do people do fast learning and adaptation? And so like one of the areas I worked on that in was like, how do you learn a word from hearing it once or a couple of times? But like the practical thing I actually worked on that project was like, how could you teach a, you know, at the time, very small recurrent language model a word by encountering it once or a couple of times? Like how reliable could you get to learn and what sort of like features of the word would it learn from different amounts of exposure, basically?

22:40Andrew Lampinen:Yeah.

22:41Elizabeth:Okay. And then when you went to industry, so you were working with language models in PhD.

22:47Andrew Lampinen:So I was, yeah, I mean, that was the only project that was on language models per se, but something I was, I worked on during my internship at DeepMind and was more generally interested in my PhD is kind of was like compositional generalization of language for grounded language learning systems. So like agents that like you give them some language instructions and then see if they can like generalize to new compositions of instructions they hadn't seen during training um and so like i guess i've been interested in various aspects of language relating to classic cognitive questions and language models were a particularly exciting place for thinking about some of those issues and i guess like you know the kind of niche i found myself in partly accidentally and I think partly because of how my background intersected with the time I was in was like a lot of the work we've done I would say is like taking sort of cognitive perspectives like rational analysis and trying to use them to make sense of like why do these systems do the things they do like you train a language model on a bunch of internet data you get things like in context learning you get things like weird patterns where sometimes the model uses logical reasoning, but sometimes it uses some heuristics.

24:05Andrew Lampinen:And these are actually like the kinds of phenomena that people have studied in humans too. And so like, one of the main things we've tried to do is things like what are the minimal data properties that give rise to these kinds of phenomena when you just like, pursue the language modeling objective. And that is very close to the kinds of rational analysis arguments that have been made that are like, what are the ecological properties that would give rise to heuristics and biases, for example, if you thought that humans were like optimizing for some objective um so we end up doing a lot of work that has like at least somewhat cognitive flavors stuff i see also so since this is for a general

24:44Elizabeth:audience could you quickly explain what rational analysis is yeah so it's basically i mean it was

24:51Andrew Lampinen:an idea that was proposed to make sense of actually the things that people do that seem irrational, right? So if we have heuristics for making decisions, like, why, why would we have that? Why wouldn't we do something that's perfectly rational all the time? And the story of rational analysis is basically, is there a way that we can rationalize this by saying, well, people are not trying to optimizing for being perfectly logical, but they're optimizing for something else, like making good enough decisions with the limited cognitive resources that they have, or they're optimizing for gathering optimal information about the world, even if that doesn't mean their choices are always optimizing their sort of like outcomes all the time.

25:33Andrew Lampinen:And so like, these kind of like accounts give us a way of like, making sense of why is it that people do seemingly irrational things. And so similarly, like, for example, we've done work on trying to make sense of like, Why is it that language models like sort of mix content into their like sort of like give different patterns of reasoning depending on the things they're reasoning about in ways that reflect some of the features that have been observed in humans?

26:02Elizabeth:okay so is that what you worked on once you got to industry like how are like applying these like you know cognitive science um background and literature to how can we like investigate this

26:16Andrew Lampinen:in these like llms yeah i mean i've worked on a a lot of different things i think that's one of the nice things about industry is that you have more potential to move between topics whereas like Like, I feel like in academia, you are kind of, at least when you're on the tenure track, you kind of have to build a portfolio of work along a single theme so that you have enough coherence to make a case about why you should get tenure. Like, what's your thing, the thing that you can be known for? So I've worked on, like, everything from, like, almost philosophy to, like, interpretability and, like, training video game agents.

26:55And so like, I don't, you know, I don't think that there's one theme there.

27:01Andrew Lampinen:But definitely one of the main themes is this sort of like applying cognitive style or neuroscience style analyses to try to make sense of why a system does what it does or how it does what it does.

27:12Elizabeth:Let me see. I'm just curious. I maybe it feels different once I actually have a PhD. But right now, I think I'm, I guess I feel like I don't know enough about what I'm researching, even so that I can't imagine moving between multiple topics very fast. I'm just wondering, is it because like, you've already had these knowledge, like transfer learnings fast. Like, what is that? What, how does that, how is it facilitated in industry? Yeah.

27:48Andrew Lampinen:Yeah. So, I mean, I think I mentioned this earlier, but industry is very collaborative and we have people from a lot of different backgrounds. So I think that is maybe the, the dominant feature is that like, you know, because I work with like neuroscientists and people from linguistics backgrounds and people from philosophy backgrounds and so on, like it's, it makes it a lot more accessible to work on a broader range of topics. And those people, I think are like much more accessible as collaborators than they, than people of their equivalent level of experience would be in academia, because they would be running their own labs and they'd be very busy with that.

28:26Andrew Lampinen:So I think that's been a a nice feature of industry research for me is that it's afforded me more opportunities to move between topics I do think that like you know as your PhD goes on I don't know what year you're in I'm a second year okay yeah like I think that the Stanford psych department is pretty good at exposing you to a wide range of things and so I think like you'll feel that you have a little bit more breadth as your PhD goes on. But I think it's also something you can choose to lean into more or less. And I think there's always a trade-off. I do, you know, oftentimes feel out of my depth, even in topics that I've worked on, because like, you know, there's people who are like, the one thing I do is work on causality.

29:17Andrew Lampinen:And I'm like, I wrote like a couple of papers that touch on it. And now people want to talk to me about causality. And I'm like, I don't know enough about the details of this space. So I think there's always a trade-off, but I think the bar to making useful contributions in a space is lower than you think. And especially if you bring perspectives from another area that that part of the field doesn't engage with as much, I think it can be quite easy to make a useful contribution. And then if you have collaborators who are more experienced in that space, it's easy to make sure you're not just totally duplicating work that's been done before or missing something obvious.

30:04Elizabeth:I see. So I don't think I've asked you this yet, but so your job title is research scientist. Is that your job title? Okay, research scientist. That seems like a very broad title. What does a research scientist mean? Like, I'm guessing it's, I'm guessing it would mean different thing for everyone with that title.

30:31Andrew Lampinen:Yeah, that's pretty true. I think there's definitely scope in industry for research scientists to do everything from writing papers to being kind of a core engineer on some research infrastructure. Are at DeepMind at least also research engineers as a separate title, but the lines are kind of blurry. I think that like, for me, like the research scientist title has meant that the work I did at DeepMind was pretty researchy. But I would say that anybody working in industry will probably end up doing more engineering as the research scientist, especially as an experienced researcher than they would in academia, where as a PI, you probably just won't have that much time to do any engineering.

31:30Elizabeth:Yeah. So what if someone wanted to become a research scientist? What are the skills that you think are important to be a research scientist?

31:47Andrew Lampinen:So I do think that like engineering is important in maybe increasingly so.

31:55Elizabeth:What do you mean engineering? Because engineering is also a very broad field.

31:59Andrew Lampinen:Yeah. So I guess I mean like having experience in like programming, systems design, algorithms kind of things. Like maybe not the background of a like CS undergrad, because I don't have that. And most of the people at DeepMind or at least a large chunk of the people at DeepMind don't have that. But like having some experience with many aspects of doing programming, doing sort of systems design, doing testing. And you can learn some of these things on the job. I think I've become a much better engineer, maybe starting from my internships, but certainly during my time working in industry. me um but it's like that is probably the like dominant weakness people have coming in or like thing that blocks them from getting accepted for internships or for full-time positions in my experience if you're if they're coming from a PhD that's not in computer science is like not having sort of like the programming skills to pass the interviews and not sort of having the experience to demonstrate that they could do a good job at the more engineering aspects of the job.

33:25Andrew Lampinen:And I think that's, you know, it's something that you don't necessarily have to learn through classes, although that can help. But like, I think there's a lot of things you can learn by just hacking around with code like in your research or like on your weekends and maybe there's a lot more resources to learn about it now that you can just like ask language models questions about

33:46Elizabeth:what's going on in your code yeah yeah cool um do you um that was all of my like official questions But I'm just curious, sometimes when I'm interviewing the interview, he has more insight into what would be interesting. I'm just curious, do you have anything else you'd like to share about your journey or anything you want to expand on?

34:16Andrew Lampinen:Maybe I'll share. okay so let me say a high level theme which is that i think industry has more flexibility than academia in certain ways and i'll give two examples of that so one is that like partway during my time at deep mind i wanted to move back to the u.s because my now wife and i were long distance at the time.

34:49Elizabeth:Oh, where were you before?

34:50Andrew Lampinen:So I was in London when I started and most of DeepMind was in London when I started. But so I wanted to move back to the Bay Area to be with my wife. And I was able to do that while keeping my same job and keeping most of my same collaborations. And I think in academia, it would be quite hard to be like, I'm moving across the world, but I want to keep my job. But that's a relatively easy thing to do, at least in big tech companies. And the second example is that, so I actually just left my job at DeepMind and I'm moving to Anthropic in a few weeks. And I think that one of the nice things about industry is that it's relatively lower switching costs to changing jobs.

35:37Andrew Lampinen:And it's relatively more expected. If you're a tenured professor, people do change jobs, but it's not super often. And by contrast, changing jobs is pretty common in industry. And so it can be a nice way to, you know, explore a little more and get exposed to new ideas and new people and explore different kinds of working. And so I think that's something that I'm personally excited about this job change. I think it's one of the nice things about industry that you have more flexibility if you, you know, for whatever reason, aren't happy or think things could be more exciting somewhere else to change.

36:14Elizabeth:other than that maybe the other thing i commonly okay two other things i commonly say when people

36:22Andrew Lampinen:ask me about academia versus industry one is that related to the sort of experience levels i think one of the downsides of industry research for people who care about mentorship is that you don't it's quite hard to build the kinds of mentorship relationships you would have with a PhD student in academia that within industry, because basically there's few people who are as like who come in without a PhD. And there's few opportunities to like do that kind of like sustained long term mentorship, relatively speaking, even if you manage people, because people change companies, change teams, etc. It's less common to be able to mentor someone for like five or six years um and so i think that's maybe for people who care about mentorship over the longer term rather than just like an internship or something that's maybe one of the downsides of industry research and then one of the upsides is that i think work-life balance

37:17Elizabeth:and lifestyle are better in industry in my opinion okay that's good um okay and i just said that's good because i've heard from some people maybe it's because they're at a startup

37:33Andrew Lampinen:yeah i do think the big tech company experience is very different than the startup experience

37:37Elizabeth:and uh startups startup people work very hard okay well um anything else no i think i think that's all okay um well thank you so much again for chatting with me yeah it was great having you on podcast.

37:56Andrew Lampinen:Yeah. Thank you so much for having me. This was lots of fun.

38:24Elizabeth:Thank you for listening. If you could leave a review on Spotify, Apple Podcasts, or whichever platform you're listening this from, we would really appreciate it. This podcast has been a labor of love by several wonderful young folks here in the department. And we've been surprised by the ever-increasing reach the podcast has had. Help us make even more people excited about Psych by leaving us a review or subscribing to our no-spam, all-fun, sub-stack at Stanford PsychPod. To connect with other listeners or shoot us an email with your thoughts or suggestions at stanfordpsychpodcasts at gmail.com.

39:02Elizabeth:Thank you and have a wonderful psych day.