OpenAI Researcher Explains How AI Hides Its Thinking (w/ OpenAI’s Bowen Baker)

24:47Bowen Baker:So so yeah. So this is like when you kind of in the intro, you were saying you mentioned that chain of thought could be fragile. And I think that is a big worry in the community right now. and this goes back to our also our conversation of you know talking about like what you know is the chain of thought or sorry is the model doing something instinctively or is it like actually thinking through it actively in the chain of thought there are certain pressures or things that could happen while training that could caught that could kind of shift that balance probably and so one is you know this thing of like if you train the model to never say something think something bad.

25:26Bowen Baker:We showed in a paper earlier this last year that it can in some cases actually still do a bad thing, but not think the bad thoughts anymore. We think it probably can only do this in cases where the chain of thought isn't actually necessary to do the bad thing, going back to this question of necessity of the chain of thought. But it still shows that you know, there's actually some propensity for models to reason out loud, even when they don't need to. And you can kind of destroy that propensity with style pressures like that. And so, yeah, the model and then we call that obfuscation, like the model is like obfuscating its actual it's its thinking.

26:07Bowen Baker:And then there's kind of a lot of other reasons we think that, you know, chains of thought could become like less information dense or have yet be less monitorable. So for instance, people are worried even about just large scale RL or reinforcement learning, which is the way we train these models, kind of just in the normal way we're doing it. You could imagine that as you increase the compute there, models could eventually learn their own languages or like change the way that they're using English, the English language or whatever language they're thinking in. At the current scale, and this is something we found in our recent paper, it doesn't seem that worrying, but it is, you know, in the, in the reinforcement learning literature, there are examples, like people were working on like emergent language and emergent communication back in like, I don't know, 2016 to 2018, 19 or so.

27:02Bowen Baker:And, you know, they showed it in like very constrained settings that if you just put in models that have no language prior, they're just totally randomly initialized, they can learn some language to, to talk to each other with. um and ideally you would want that language to be understandable by humans exactly yeah yeah and so yeah so so but this might happen where like the model would like learn its own internal language of how to think that's maybe like a bit more efficient or compressed than the english language and it's like then it's harder for harder for us to understand ourselves or a different model to understand that hasn't like learned that same exact uh yeah language uh specific language okay You mentioned in your research that you found longer chains of thoughts to be generally more monitorable.

27:52That's a mouthful. Are there things that you're doing in an effort to get the model to show more of its work?

28:00Bowen Baker:Yeah, so kind of what we've talked about up till now has been maybe like, we talked up until like the thinking that was done and work that was done until maybe like mid two thirds of last year. And, you know, we knew it was useful. We had been monitoring these big reasoning models. People were in the research community were starting to get worried that it was fragile, that it could be like, you know, this could go away, but they thought it was important. So they wanted to work on it. And so at least to me and I think many others, if you want to preserve this property, like the first thing we had to do was be able to measure it.

28:39Bowen Baker:Because I guess if you think back on the conversation, I never said any kind of like metric for how monitorable a model was or like how to eval. It was all just like vibe space. Yeah, you can't measure. Yeah, exactly. Yeah. So building evaluations where we could actually kind of start to quantify how good or like, sorry, how often these models verbalize their thinking was the big push that we made in the second half of last year. Because, again, yeah, to be able to preserve it, you have to be able to measure it. And so, yeah, we built this suite. I think I would call it like a good starting point of evaluations.

29:19Bowen Baker:So there's definitely much more work to do. And I hope the community kind of continues to build upon these and share evaluations with each other. We're working on open sourcing hours to share with them as well. But once you have the ability to quantify this, then you can start to actually make, you know, like interrogate different parts of your training pipeline. You can interrogate different decisions of monitoring. So like you said, like if I have a model that has a longer chain of thought, it's much we found that it's actually, you know, generally it reveals more information in that longer chain of thought and then it's easier to monitor it.

29:57Bowen Baker:And so that was like one of the one example of a thing we you can start to actually like make statements about once you have these evaluations. Right. You also mentioned a monitorability tax where sometimes you might accept slightly worse performance to keep the AI systems more transparent. Is that something that you I think you're advocating for that, right? I am not advocating. I mean, probably. I would say probably I would advocate for that. Yeah. Sure. Or sorry. Yeah. Definitely in any hypothetical scenario where you're deploying a dangerous model or a safe model, I'd say you should probably deploy the safe model, even if it's less difficult.

30:34Even if it's a smidge slower or something. Yeah. Right. Exactly. Yeah. Well, when would paying that tax actually be worth it, I guess, is the is the question.

30:43Bowen Baker:Yeah, there's probably many different types of taxes. So for instance, like we talked about the latency tax already, like even if you just deploy the model you want to deploy, but you want to monitor it, you now have to have a slightly higher latency probably to monitor it and before it takes actions or gives outputs to users. So that's one kind of tax. the tax we talk about in the paper is that we found we did a experiment to investigate the effect of the pre-training compute size so you know the gpt paradigm has been increasing the size of the model while increasing the size of the data and then on top of that we've like now have the rl paradigm which like you know we take those pre-trained models and we we we do another procedure to make them really good at reasoning.

31:35Bowen Baker:And so we wanted to investigate, you know, how does monitorability scale with pre-training size and pre-training compute? And so we took a series of increasing size models and we measured their capability and their monitorability at like different reasoning efforts. And so like, you know, models can generally be set to have like a lower or a higher reasoning effort. They'll think a bit shorter or a bit longer, respectively, for each of those. And so reasoning models, generally their capabilities like on math problems and coding problems and things like that, they get better as they think for longer, they'll be able to like solve more complex tasks.

32:12Bowen Baker:And so a small model thinking for a very long time can oftentimes get to the same answer or max the capability of a bigger model thinking for a shorter time. Okay, wow. So that's, yeah, this model is really tight. There's like a couple more steps in the thing. I can just go through it and then you can. Yeah. Sure. Yeah. So, yeah. So you now have like these like capability matched points of, you know, small model thinking for longer, big model thinking for shorter are like roughly the same capabilities. But we found that again, going back to your previous thing, we found that models that think for longer are more monitorable.

32:48Bowen Baker:They like reveal more in their thinking. We found that the model thinking for a the smaller model at the same capability level was more monitorable because it had it was it was thinking longer trace yeah yeah that makes sense um and so then that's the monitorability tax or sorry but um sorry the then the tax aspect comes in where these small models thinking for longer actually were using at that for like matched capability levels used more inference time compute than the bigger model thinking for shorter. So which one would be more expensive, I guess, with like a smaller model thinking for longer or a bigger model thinking for shorter?

33:29Bowen Baker:In this case, the smaller models, I think, basically always cost more because the relative, you know, decrease in inference compute per token in generated was less than the amount of additional tokens you'd need to reach the same capability level. And so overall, that small model with a lot of tokens was costing more compute than the big model. And so that's kind of like the monitorability tax. It's like, oh, you could just spend more compute on inference time, but like get a more monitorable model out of it. Okay. That's tricky. Yeah. What are the warning signs maybe that a system's reasoning is becoming less monitorable over time?

34:16How would, is there a way to recognize that that's happening?

34:20Bowen Baker:Yeah, well, we, I guess we hope so. I mean, we hope these evals are the one or maybe the way to do it. So we have a few different ways of kind of like tracking monitorability. We like divided them out into three archetypes. Uh, and we hope that like, you know, if we make a broad enough suite of evaluations that test monitorability in different ways that when, you know, a model does start to like become obfuscated or verbalize less for whatever reason, we'll see one of those evals start to go down. Um, and we did kind of show a proof of concept of that in the paper where we directly deliberately trained models to obfuscate by putting some kind of style like pressures on the COT.

35:07Bowen Baker:And we then did see some of our evals decrease as like a proof of concept that, you know, if a model, if this kind of happened in the wild naturally, we'd maybe be able to catch it. Wow. Right. I guess it makes me wonder, like if you were to like, let's say like the next scaling paradigm, whatever, whatever the next step up from here is, would that model be incredibly hard to to read its traces? totally possible i guess you know if i knew what the next scaling paradigm was a much much richer man well i guess i i guess i should clarify like let's say if we go off of the previous scaling scaling paradigm which is which is just size and data right compute and data if you were to scale up to like the next the next size up of compute and data if that were even possible um would would that model not think for very long and then would that then make it It makes me think that that model's thinking traces would be even harder to interpret because it's, you know, following the logic of what you've proven so far.

36:10As it scales. Yeah, it's a bigger model and it's thinking for less time to get to the right answer.

36:15Bowen Baker:Yeah, I think that is definitely a worry. We've also like, I mean, so generally in our current experiments, we haven't necessarily found that to be true to a significant extent. I think that this is likely because, you know, we are always improving our ability to train even, you know, even a small model. It's not like we have our GPT-4 size model and that's the best GPT-4 size model anyone could ever train. We're like improving our algorithms, our data, you know, no one's sleeping. Yeah. And so I think that, you know, a better model also kind of like has more clear thoughts sometimes, if that makes sense.

36:58Bowen Baker:Like a small model, if you look at its thoughts, it's it's it sometimes feels a lot more gibberishy than a bigger model. And so, you know, that is maybe the reason why we could expect this to, you know, at least not fully this trend to not fully happen. But I think at the limit, it probably should happen. Like bigger models will just verbalize less for the most part yeah i guess that makes sense you tested whether uh something that interested me was that you tested whether penalizing bad thoughts during training would stop that kind of misbehavior i wonder if you could tell us a little about what you found there yeah um so what we did were so when we this was kind of earlier last year and we were worried that people would kind of like start applying some like various pressures to the chain of thought because no one had kind of like thought through it yet.

37:57Bowen Baker:And we were like hypothetically worried that, yeah, if you tell it to not think bad things, it could still think bad things in its like activations and then do a do a bad thing afterwards. So we did this experiment where we, you know, trained two models, our baseline that has like no, I don't know, we can call it like thought police or something. And we're not present. and then another model where the thought police were present. And the model with the thought police present actually did like do less bad things when you because, you know, which makes sense, like our models are like oftentimes very causally, like they're causally affected by their chain of thought.

38:35Bowen Baker:If they don't think a bad thought, they might not actually do a bad thing. And this has been shown in other papers from other groups as well. But eventually, once it started to learn to obfuscate and and like still do bad things, but while not having bad looking thoughts. At that point, we found that like, you know, while it still did a bit less bad stuff, you would have preferred to have a model that was like openly doing bad stuff and thinking all the bad thoughts and using a monitor. So like, for instance, you could catch more behavior, bad behavior, a higher percentage of bad behavior of the model that wasn't trained with thought police using a chain of thought more than you could when you like had done this whole like thought police training.

39:20Bowen Baker:And so you were better off at the end of the day, like not doing it. So speaking of open windows into the mind, you know, we're looking at the thoughts here. You also published a paper recently on weight sparse transformers where you prune the model to make its internal circuits easier to see. How does like, I guess, mechanistic interpretability and chain of thought monitorability coincide? How do they work together? And what are your thoughts on that? Absolutely. Yeah, I was on that paper. But if you look at the acknowledgement or whatever the contribution statement, I was just the manager of Leo.

39:56Bowen Baker:Leo is like much more the experts in that. But I appreciate the disclaimers. I wouldn't take credit for that paper. But I think that they are like in general mechanistic interpretive. I think the circus varsity work is awesome and like a very preliminary kind of like early sign of life for a type of model we could train that was like natively more interpretable in its activation space. And I'm excited for people to kind of continue pushing on that and more broadly the mechanistic interpretability field. Historically, it's been pretty hard, like mechanistic interpretability. You know, we haven't solved it yet and people have been working on it for a long time like people could you at a high level explain to our viewers like why it's so difficult um yeah i'm not like an i would say i'm not like an expert in this but my take for why it's been so hard is that similar to i guess like one way to break out the difference first between like chain of thought and mechanistic interpretability is mechanistic interpretability is more like doing a brain scan of all your neurons and activations in your brain, and then trying to make some predictions about what you're going to do from them, which neuroscientists do do.

41:15Bowen Baker:And it's not like impossible, but it's hard. You know, there's still like, you know, we haven't, we can still have a hard time doing it for mice and things like that. Right. It's not a flawless process. Yeah. Yeah. And it's just really high dimensional. There's a lot of is going on, you know, you're trying to find in like these like billions of neurons, you know, which pattern amongst them is the reason why you like do a bad thing or the reason why you like walk or, you know, any or twitch your muscle. There's so many things they're doing all at once. Whereas, you know, the chain of thought interpretability thing is a bit more like reading your inner monologue, which is much more high level, much like, I guess, like lower information density.

42:00Bowen Baker:It doesn't have everything there, but it does seem to have like the most high level relevant things to the model's actions, which is why it's been so successful in comparison. And like, but again, like is like mechanistic interpretability does not have the issue of like the same issues of fragility that chain of thought interpretability has because the activations aren't going to go anywhere. The information should all still be there at the end of the day, but it might eventually not be in the chain of thought, depending on how everything goes. That makes sense. I have a question I've wanted to ask a safety researcher for quite some time, and it is, does the concept of open source AI concern you in any way as someone who's focused on safety and moderability, and why or why not?

42:52Bowen Baker:um i'll yeah this is so i would just give my own opinion i i wouldn't that's fair yeah yeah just just as bowing yeah yeah yeah but um yeah i'm pretty worried i would be pretty worried about open or i'm i'm worried about open source i think that um you know if you had like there's a reason why we like try to control like certain types of information from the public um and have or like have controls over like weapons and things like that. And if you think a model could be utilized as any kind of harmful weapon like thing, you know, whether it be making a bioweapon is a big thing people are worried about.

43:32Bowen Baker:Cyber security or cyber attacks is another big thing that people are worried about. I think just kind of having like a downloadable weapon online sounds pretty bad to me like so um yeah and i think it's probably pretty hard to put in like the proper guardrails on an open source model because someone can like just fine-tune those guardrails away once you've released the weights and you know yeah that's a thing i have wondered for so long and i really appreciate you sharing your personal opinion there uh because i i always thought like Like at its core, I try to be a good open source promoter. And at the same time, I'm like, you know, you kind of lose the button, though.

44:17You don't have the off switch anymore. And I mean, once it's out, it's there. You're never undoing that. You know, you're not going to go delete it from every computer on Earth, you know. And it's just a subject I've always found really interesting. And then the whole safety, interpretability, monitorability, all of that's really, it's an interesting area of AI and exploration. And I really admire the work you guys are doing. Thank you. Yeah.

44:45Bowen Baker:I will just add a tiny piece of nuance in that like the current open source models do not seem dangerous. And people have done like the worst case fine tuning scenarios for the model. At least we did for the models we released. and we said okay like even if someone super nefarious went and fine-tuned this model to like you know develop bioweapons they just couldn't really do it like this model is not smart enough and so i think in that regime it's totally fine like there's not like the risks aren't high and and it's like useful and it's like super useful for the community for doing research for making products whatever they're doing and so in that i think i'm i was more thinking in like the um you know, like the, the highest power thing of the future.

45:27Bowen Baker:Yeah. Yeah. Right. Makes sense. Do we really want this on a flash drive? Kind of, you know, I get that. I get that. I love the OSS models, by the way. I've used both. Yeah. You worked on that, right? You at least contributed to the system, the model card for it, right? Yeah. I was just involved in some decisions around like whether to leave the, actually like one interesting thing with the open source models that we like didn't put any pressure on the chain of thought. We just like let it think whatever, you know, if it's going to think something offensive, we let it do that because we wanted it to be, you know, like we wanted developers to be able to, you know, do the same type of chain of thought monitoring that we do if they wanted to in their product.

46:07Bowen Baker:And then also have like a model that is close to our internal models for external safety researchers is kind of a useful artifact to have out there so that when they do research And if they do it on our models, maybe we're a bit more convinced that it would apply to like the big, there are bigger models that we have internally give us a bit more confidence to go and implement it and try it ourselves. So it's a lot of benefits to that. Yeah. Okay. We have a question just that's kind of for fun, which is how does research work at OpenAI? Like, do you all pitch to Sam and then he gives you the thumbs up or Jacob or whoever's in charge of the science there?

46:46Or do you get it assigned to you? like how do you decide what your next or you just grab an idea and run with it or yeah yeah do you guys have to fight for compute like how does that work i would say like a bit of all of that it

46:59Bowen Baker:really it's like i don't know there's no there's no it's it's not extremely formal um every team i think works very differently um teams like mine that i run are a bit more like we don't really know okay like when we started we had no idea what we're doing we're like trying to figure out like what the research program was, what mattered, what didn't matter, what even the definitions of the thing we were looking for was. This term monitorability, we had to argue back and forth of what is the... Because people talk about faithfulness a lot, which is a related concept. We got rid of that and went for monitorability as the more practical thing at some point.

47:37Bowen Baker:And so anyway, we were very much more in the trying to figure everything out, do a bunch of like research and we do and we still do kind of like do a bunch of we kind of give people a lot of leash to just do research that's interesting in this space because it's so new and we know so little I think in cases where things are more defined and you have products that are like you're you're making there's probably a more clear path forward for a lot of things and so then research there becomes a bit more like defined or something like pre or predefined. I know Sam cares about chain of thought monitorability, but I think he's pretty he's pretty busy.

48:19Bowen Baker:So, you know, I don't go to him with every idea and ask for his OK. So, yeah, that's fair. That's fair. I imagine he's a busy fella. Yeah. So you have a little bit more leeway there to say, OK, this is important to us. We need to go. We need to know about this. Yeah. But generally, everyone's super smart and has really good opinions. And so I think it's it's like not that top down. It's a lot of it's very bottom up. Yeah. And like, you know, at the end of the day, it's like you kind of want your manager to approve of what you're doing probably. But generally everyone's like open up, open to discussion.

48:53Bowen Baker:And if you pitch a good idea, they're going to be like, yeah, that sounds great. You know, let's do that. Go for it. Yeah. That's awesome. Makes sense. Okay. Last thing on, I guess, monitorability and the reasoning traces. So when we as a user see traces, I've always wondered, what is it that we're actually seeing? Because it's not the actual train of thought, unless you're looking at the open source one. So how does that work? Is there a model interpreting the traces? Or is it just like making it up? Like, what are we actually seeing? Yeah, absolutely. Great question. So yeah, we do show users like some of the thinking, but it's from a summarized, or sorry, We have a summarizing model that takes the chain of thought and like summarize it, it summarizes it in a safe way that's, you know, kind of like adheres to all of our product policies that the user can see and is more digestible.

49:44Bowen Baker:I mean, it sounds like you guys have looked at the open source model. Like when you ask it a hard problem and it thinks really long, it's like a lot to read through. It's got a lot to say. Yeah, it's a lot to say. And you don't want to read, like maybe it's fun the first time, but the second time you definitely don't want to read through like pages and pages of it thinking stuff. and so i think even from a product this is just my opinion like i would guess from like a product uh standpoint the summaries are probably preferable um i have no idea if they are but i just actually yeah because i look at all of them right i look at you guys i look at gemini i look at anthropic and i think that open ai is definitely more it gives me more information about what it's doing than gemini says for sure and for me as a user the main thing i want from them aside from occasional entertainment is I want to be able to see like if something went the wrong direction if I can understand where it went wrong I can understand maybe how I prompted something wrong how I could have asked the question better and being able to see that that feedback lets me know oh I just I just gave it some nonsense here and it it winged it and that's that's on me that's that's not a model problem that's a Corey problem you know and sometimes that's the kind of shows Yeah, the flaws in your own thinking, actually, which is kind of interesting.

51:01It does. It does. I'll never forget the first time watching it, though. We were, a buddy and I here at work, when a one first released, we were watching the live stream on our computers and chatting via Slack. And the second one of us got access, it was like, okay, quick, hop on a call. And we jumped on it, started throwing things at it. And as we watched it think, I just remember thinking, this is like nothing we've ever seen before. This was wild. And it's crazy how in not much more than a year, heck, in a matter of a couple of months, frankly, it was what you just expected out of a model now.

51:38And it moves really fast in this space. And it's really interesting. And I can't imagine on the research front how fast you feel like you're moving.

51:46Bowen Baker:It's all, you know, if I could do it all myself, I'd maybe have it move a bit. I'd wish it would move a bit slower. I kind of agree with you. I mean, it's exciting for us, but also it's like, give me some time to process. Yeah, Grant and I on the week of Christmas were like, you know, it'd be really great if all these companies would just have like a gentleman's agreement for 14 days here so we can all go get some sleep. It kind of happened. I'm sure they would all appreciate it too. People did some stuff. Oh, I did have one last lightning round question for you, which is, are you, Corey and I debate this.

52:19Are you thinking on always or depending on the task? So for me, I'm always every task, always thinking is on. Where do you land on?

52:27Bowen Baker:And I'm a router guy. Oh, yeah, I'm definitely a I think generally I'm a router guy. Like I like it to come back fast if it's like a thing I know is simple. Sometimes, though, I think the only time I personally use thinking is I've asked it a question that I thought it should be routed to thinking or like I know that it should think more. And then I go and turn it on and like kind of ask it to, you know, think really hard about a thing. So like if I'm like asking it some advice about some like math thing at work, then I'm like thinking only definitely like maybe the pro mode or whatever. But don't you wrap this to instant, you know, better than that.

53:04Yeah. And then you go kick it on. That's kind of my approach to is I didn't for a while. It took a while to get used to to the router. It was like, oh, I don't know. and uh over time i just one day realized it was mostly just working for me uh usually there are there are times like you said where i will go kick on thinking uh but for the most part it it does the job fine yeah yeah the models are getting pretty good all around so they are i was we were talking about that the other day about how they're just they're kind of all good now like like we don't there aren't really terrible models out there like there were a couple of years ago.

53:44Bowen, thank you so much for joining us today, man. It's been a lot of fun and really appreciate the work you're putting into a very important problem.

53:52Bowen Baker:Thank you. Yeah, it's been a blast to chat about with you guys. And yeah, thanks so much for having me on. Bowen, where can people find your work and follow what you're doing next? Yeah, I mean, our, you know, the OpenAI blog and now the OpenAI's Alignment blog will be a place We continue to post a lot of our things. You can probably find me on Twitter, but I don't post that much, so it's not that useful. Yeah, I mean, we'll be publishing safety work is like a good way to to adhere to the mission. So we'll be trying to publish everything we can to help everyone make their model safer. So that's awesome.

54:30Well, thank you, Bone. We appreciate it.

54:32Bowen Baker:Thank you, guys. Well, hey, if you're watching today, please take just a moment to like and subscribe. We really appreciate it. It helps us continue bringing you fascinating guests, the people who are building the AI and working on it. And we want to keep doing that. So please make sure you give us a follow. Go sign up for the newsletter at theneuron.ai and join a whole bunch of other people who do that too. And on that note, we have nothing else for today. So farewell for now, humans. We'll see you next time.

55:07Thank you. Thank you. Thank you. Thank you.