OpenAI Codex Tech Lead On How His Career Grew And How He Uses Codex | Michael Bolin

25:01Or is this kind of also leadership buy-in? Hey, this is going to be valuable. And you got like a doc.

25:07Michael Bolin:Yeah. Yeah. I think, I think both cases, it was certainly like bottoms up. I don't think so, you know, things like react and PyTorch, like those are like the, the big success stories, right. Where, where the value back to the company is like unquestionable. And then there's this like longer tail of things where, um, I think depending on the economy and other things, the managers get grumbly if they feel like their engineers are like doing too much open source or things like that. So it's almost, almost always, um, I feel bottoms up. I would say that's often the case. But it wasn't met with a lot of resistance.

25:46Michael Bolin:And you usually get a good conference talk or two and a nice blog post. And those blog posts do, I would say, pay dividends over time. Right. Drawing, recruiting. Yeah. They have a longer shelf life than I think people realize. Okay. So you got your EA promo at this point. And I imagine the expectations maybe in your mind are kind of going up a little bit. And now you need to go and find a E8 problem, I guess. So what'd you do after you got voted? I think that's where I did a, I got a little over my skis and I tried to help with web speed, I think is what I did. Because again, it was a thing that was like a really big problem for like the, just the load time of facebook.com was not in great shape.

26:34Michael Bolin:You know, the architecture was a bit stale. and it was just the problem was so big and I didn't have I think the you know a lot of people who had worked on web at Facebook had worked on it for like a long time and like I had you know put myself in mobile and dev tools I actually wasn't in that world and I remember I was like ah you know what do we do I mean actually I sat down with another person we started like compiling V8 from source and trying to see if we could change the way that we generated JavaScript, if it was going to be friendlier to VA. We were just trying to whacky things, none of which panned out, by the way.

27:15Michael Bolin:And I just, it was, there's different people who are good for different projects. And I think that was just not the one. Like I'm better in a project that involves writing a lot of code from the beginning. And I think a project like that was more about looking at data and talking to a lot of people. And like, I just, that's not my strength. I think you mentioned that at this point in your career, the idea of a hero quest. Yeah. What's that mean? The idea that, you know, that it's like, you know, it's embarrassing because there's like something about egos. This idea that, you know, there's this like, you know, Gordian knot and all these engineers are like, if only someone would come in and like solve this, you know, engineering problem.

27:59Michael Bolin:I was like, I know JavaScript right now. Okay. Just come in that way. And, but I didn't, I definitely did not, you know? And I think that it's been an important learning and I've had to relearn it like at least one other time that, you know, I can do a lot of things, but there is a smaller subset of things that I genuinely enjoy doing and that I'm going to be a lot more successful, right? If I stay in the things that I genuinely enjoy doing. You know, I try to expand that over time, but, you know, we don't all have to be the best at everything. Right. I think that's if she accepted and, you know, embrace it.

28:39So then how'd you find a problem after that? Like, what was the next day?

28:44Michael Bolin:That was, I think, a bit of luck as well. I guess, you know, we had, we had, sometimes we'd have these little summits, uh, with like smaller groups of engineers about like brainstorming what's gonna, you know, bite us in the future. I mentioned that. The repo keeps growing. We're going to have some scaling problem at some point. And a person or my manager, I think he became my manager, Brian O'Sullivan, he decided to get some people together to work on making a virtual file system to try to get ahead of that problem. And so I got myself and Adam Simpkins and Wes Furlong, both tremendous engineers.

29:27Michael Bolin:I actually thought I was the worst engineer on the project for quite a bit. You mentioned a virtual file system. On a high level, what's the benefit to a company like Meta if you had something like that? If you have bought into the monorepo philosophy, put all the code in one repo, most things that people do need only a subset of that repo. They have to look at the files at any point in time. and um so the idea is that uh you know you design all your tooling around those virtual file systems that when you say uh clone the repo or then you update to a different commit or anything like that you don't actually have to like write out every single file in the repo on disk when you make that change which is which is the default in your traditional file system because that is going to grow proportionally to the time with the size of the repo right so at some point you're you know, going to be very sad.

30:22Michael Bolin:And, and, and so, you know, there's actually kind of two, at least two parts to it. One is building this virtual file system that's like, hey, I know the users at this commit, if, if the operating system asks me for the contents of the file, like I can go get it, and it will appear like they had actually, you know, laid out all the files. The other part of it, which is the part that maybe I was actually better at, I think, is then try to anticipate, well, we have all these tools in our tool chain that are used to just reading all the files. You just rip grab, it just reads everything, that sort of thing.

30:57Michael Bolin:And how do we start changing also our development flow and our tools such that they are designed with the virtual file system in mind? Because if you just materialize all the files with your tool, now you've actually just lost all the benefits that you've made. So on a high level, it's basically just lazy loading a huge file system. So it's more efficient because you don't need to do everything at the beginning. Yeah. And so that part of this project that you said you're better at is integrating everything into this on top of that primitive. Yeah. Yeah. So one thing I did actually with Hanson Wong, who's here, actually, I'm on the Codex team with me now, was I was like, oh, OK.

31:40Michael Bolin:You know, the traditional way you have, you know, you want really fast file search in your IDE and your editor. And, you know, the way most of these things do is the first thing you do is they walk the entire file system to find out what the files are. And I'm like, well, that's going to be a serious problem, right? It's going to undo all the benefits. And then it's one of these things is like not, you know, first it was like, okay, how can we implement file search that's not going to undo all the benefits? And then it was, how can we do it even better than how it's done today? And so we ended up building this file system called Miles for my files.

32:18Michael Bolin:And, you know, on like kind of like a cron job, it would index, it would ingest all the new commits that had come in on, you know, on trunk and keep track of like which files had been added or removed. So just like the names of the files, not the contents, because all you need is the names. and um and then and then uh hansen had some clever ideas about how we maintained that index um such that we could support fuzzy file matching so instead of just substring matching right like you type you know like a if you have like a camel case thing you want to be able to type just maybe the uppercase letters or just your spelling's bad or what have you um and so it came up with this really uh interesting way to represent uh you know kind of all the files that had been seen at some point and then some bits to represent, you know, if I'm at this commit, was this file present or not present at that point in time?

33:11Michael Bolin:And then also when you ask, I mean, you sent a query to this thing. So you'd send like what commit you're at and like if you've added any files locally or removed any locally. And I think we got this, you know, like, you know, over a million files and like, I don't know, like 10 to 20 milliseconds or something like that. so that you would be in your editor, you know, typing. And so that was way faster than, I think, than, you know, like what Xcode or sort of like MVS code or something like would give you out of the box. And so that was, you know, so like it was solving a problem for Eden, that was the name of the file system originally, the virtual file system, you know, in anticipation before it was even ready.

33:54Michael Bolin:But then it was so fast and it was available as a trip service internally. Like people started using it for all sorts of other things. I think when I left, like, I don't know, I think there were at least like 30 servers running miles, like spread around the globe. So clearly that was more than just people like personally searching for files. And then you did that much capacity. Interesting. Yeah. I mean, when you talked about, I guess, the implementation details, most people, they don't use leak code on the job. But that sounds like, I was thinking about the, I don't even know how you input that.

34:27Is it like a tree, like a try thing?

34:29Michael Bolin:No. So that's what's funny. Yeah. This was pretty cool in that we had kind of like two parallel arrays. One was like the file contents and one was maybe an integer into that index. I forget. And then we had a 64-bit mask that was like, it was like, so you had like 26, lowercase, 26 capital, 10 digits, and like maybe dash and whatever. And it was, every bit was set if that character was used at all in the file that you were searching. And so the first thing was we could like blow through that list and, you know, exclude a bunch of things like right off the top. But we had also like very designed, like all these arrays were in parallel to each other so that for like cache wise, we knew it would be very efficient for the CPU to like read memory, you know, linearly.

35:31Michael Bolin:And then it led itself to, you could just partition that array. It, you know, led itself to parallelism. um so it's really yeah i should really probably write it up at some point because i just that is really cool it was it was funny because it wasn't just kind of like out of the textbook that's that's for sure i understand that um yeah you worked on eden and then obviously there's miles as well um and then these eventually led to another promotion and um but prior to that promotion there was some learnings you might have had about influence and conflict and or so maybe if you want share that?

36:05Michael Bolin:Yeah. I mean, that's one of the things I guess about being an EA who primarily writes code, right? There's other people, I'd say actually the majority of people about level or higher are not writing code, right? They're actually kind of exclusively spent doing more like influence or working across teams, you know, writing the big Google doc and getting everyone on board and all month sort of stuff. And so, you know, as an E8 trying to have that level, like, you know, that level of impact that you're expected to have, it's like, oh, it's hard to do that just writing code. So it's like, I need to spend at least some time probably, probably influencing other people.

36:50Michael Bolin:That would probably be good for me. That would probably make my manager happy. um and i think uh you know sometimes you're just so uh confident in in some uh insights you have that or certainly i was that i would just i i just came down like way too hard i would say and uh and that did not go well for me and yeah and so like like a promo got delayed because i was very excited or anxious about uh when microsoft acquired github and uh oh yeah because new collide was built on this um you know primarily github technology and i was like that's gonna go away because vs code is gonna not make them be a project anymore which was true which did happen and i was just so uh anxious that that this was like now a risk right and i was pushing people But, you know, people were I didn't really account for the fact that people were happy with what they were doing at the moment and like didn't want their cheese moved, you know, like right out from under them.

38:03Michael Bolin:And so, you know, I got like a little bit of a talking to. I had to wait a little bit for promo and things like that. But, you know, I actually got some coaching, I think, after that point. I can't remember the exact chronology to work on that, like specifically. What's the number one thing you learned about coaching? I would say for me is like, I think I'm more aware of like things that trigger me, like the technical decisions or what have you and recognize when that's happening and be like, okay, let's not act in that moment. Or, you know, if I don't think I can have the conversation, like, or I'm not in the best place to have it, like maybe I go talk to the person's manager instead rather than like, you know, like bull and China shop to the engineer and be like.

38:50Michael Bolin:So I have this thing. I'm thinking about it this way. I'm kind of riled up about it. Like, help me. Like, how can I, you know, work with your team or whatever that's happened to you? Well, it's interesting to me in your career because the promo got postponed and you were seeing that VS Code was going to come up and the thing that Newclad was built on was going to go down. And you were actually right in hindsight, too. Like in the future, that is what happened. and you were battling for what was right, yet you were told, calm down a little bit. So, I mean, what are your thoughts when you saw things play out and you go, actually, I was right the whole time?

39:33Michael Bolin:I did have some conversations like about a year later and I was like, hey, can we balance the ledger a little bit? Like, it didn't be pretty hard for that thing and it was kind of good and we worked it out, I would say. Okay, so I guess the learning is just how you went about it, not what you were like. Yeah. Yeah. Um, I would say that's true. You seem to have a great time at Meta and eventually you left. So what was the thing that drew you to OpenAI? Yeah. I mean, um, a number of things. I mean, one was, uh, so I, I interviewed at the end of 2023 with OpenAI, I believe. And, um, when I had spent 2023 at Meta, trying to do LLM-based developer tools, right?

40:19Michael Bolin:There's, you know, we had our own light-up version. Even Metamate is like a little version of GitHub Copilot, right? That one, we didn't do like a paper and a talk on that one. It was Code Compose, right? That was Code Compose, yeah. And, you know, we, like now, there's a lot of enthusiasm around, you know, delivering quickly and pushing the boundaries of that sort of stuff. And, you know, I mean, truthfully, I mean, you know, you get feedback on some of these things. It's like, why is this not, you know, GPT-4? And I was like, well, we're on like Lomba 2. It's not the same thing. And, you know, I was not a researcher, right?

40:59Michael Bolin:I was like, but I, and I, you know, just wanted to build the experiences, right? And that sort of thing. and um and so so that was that was one thing was that like i wanted to build stuff and i wanted to go to the place where i could actually build with the best model um you know two again was um similar to like you know google originally like seeing the people who were coming here to open ai and i was like well those are people i really respect i was like those are i could actually work with more senior people you know like meta eights and nines at open i felt like that i could where i was and you know third was just like this also seems like and it has been just a very special place at the point in time so I told a lot of people I was like you know I felt like this would be like the most similar to starting at Google in 2000 so not like not to 1998 but like let's say 2000 right like they kind of got some footing and got some you know product market fit and so like just as a personal level that was just a very exciting thing um and um you know and the last one was that uh funny thing is that when i i really enjoyed calendar because i it was consumer and i shipped to a lot of people i went to facebook because i thought huge consumer place ended up not doing consumer at all right ended up doing developer tools but like you know my users were my friends at work it was just like 20 000 people not like a billion people but it was good enough for me at the time.

42:28Michael Bolin:And then, you know, thinking about open AI and then the chance to actually, you know, come back to, to consumer, um, or at least like have a large user base. Right. And so now working on codex, um, you know, I wonder if there are like over a million weeklies or I forget how much it is right now. Um, and, and just keeps growing like, like, you know, hockey's there more like vertical line than hockey stick, even, um, that, uh, you know, So it's like way more than the 20 ,000 to 40 ,000 developers, you know, it could affect it in meta. Yeah, absolutely. Dev tooling for the industry almost. Yeah. Meta, when I think of the company, it's kind of engineering driven, like engineers are king and queen, I guess.

43:13And it kind of like drive everything. It's very bottoms up from that sense. And I feel like a lot of the lab companies are also like that on the research side. So rather than engineers, the first class citizen, it's like, let's make sure the research goes well. And for good reason, right? Like, that's also part of why you came is the models are good. But as an engineer, you mentioned, you know, you weren't doing research. What are your thoughts on that research-led culture versus engineering-led culture?

43:41Michael Bolin:I mean, it's certainly an adjustment, right? I mean, I think of anyone who comes here and says otherwise is a liar. But, you know, if you've been at like the Fangs or whatever. um and uh but you know they are you know you talk about impact right which i think is important like a real like you genuinely you know mean it um like i you know i love the work that i do on codex and the harness and i think we do a lot of very meaningful things um but you know if the model weren't very good it wouldn't really matter what we did you know on the harness right and so

44:17Michael Bolin:So, you know, that's how it is, I would say. But, you know, I feel really great. Like, you know, we sit right next to the research team. We're really close with them. And so that relationship that we have, that we get to co-develop the thing. I mean, that was another reason, you know, leaving Meta. I mean, was that for in the LM space was like, you know, I want to build the product with the people who are building the model so we can do this thing together. I mean, you know, maybe you could do that there sort of, but not. Anyway, the impact was not, you know, quite the same. So you mentioned, I mean, when you got here, it sounds like you were working on Codex and starting that project up.

44:58So I understand with the initial launch of Codex CLI, it was not exactly what you hoped for, like in terms of how it was received, but later it kind of really all came together. Can you tell that story?

45:13Michael Bolin:Yeah, sure. I mean, it's been a wild ride. So Codex CLI, we launched it in April 2025. It was kind of like this one more thing moment at the end of the 03, 04 mini live stream. and uh so we demoed it live we open sourced it you know got three po at that point uh a lot of people tried it out you know everyone was excited to try a new um coding agent you know and it was and it was pretty good but it was it was pretty rushed um to get it out the door right there is um which in some ways was good for engagement because now we're open source we were getting pull requests you know coming in all over the place and uh i forget you know we were i feel like we're like 10 to 20 000 stars in like a week or two or something like that so like that that part was fun and um you know there were people who did like it you know it's like nobody affected or anything like that but then um but that i i think we didn't we weren't maybe quite staffed to like really drive that the way that we needed to um because you know we're we're trying to cover multiple things as a company and so you know just a month after that and you know team of like seven engineers and I forget how many researchers launched Codex web or cloud web where you can use Codex in a container or you could just, you know, kick off a new thing from your phone, which is super cool.

46:33Michael Bolin:And, and there, that was like a more well-staffed effort. And I, and I think, you know, that one is still the, I think the long-term vision is, is correct. But I think with a lot of this stuff that we've seen, you know, you have to bring the users with you and, And I think that was just like a little bit ahead of its time. And people were still actually more into the local coding agents. And so we kind of, you know, we saw, you know, the web product, like there's a big adoption initially. It wasn't, you know, as sticky, I would say, as we had hoped. And then through the summer, you know, both products still kept, you know, we're working on both.

47:15Michael Bolin:But then somewhere that mid-summer, again, like I said, this moment, local agents are still, I guess, the stronger product market fit. But again, I still think in the limit personally, right? You're going to need more machines than just your laptop as a place for agents to run. And so we shifted quite a bit over the summer. So we brought more people onto the Codex CLI. Yeah, a few of the kid banks. Bring more people on. GPT-5 was going to come out, and that was looking really, really good. And I think I was personally excited about, because I prototyped it a couple times before, was that in addition to the CLI, then we also had enough staffing.

47:58Michael Bolin:We also started working on the VS Code extension. And I felt actually very strongly that the terminal is good for a lot of things, but it has a lot of limitations always. it's just, it's just like you make a lot of compromises to make, you know, a nice UI in the terminal. Whereas VS Code, we didn't have to make quite as many compromises. And so I think like August, I think it was a crazy month. I think GPT-5 came out. I think we released our new refresh terminal UI. Also the GPT open source model came out. We supported that in the TUI as well. So that was really cool that we had, you know, an open weights model and an open harness.

48:37Michael Bolin:and then later that month the VS Code extension came out and so we were just like shipping like crazy and that's really where that confluence of things I would say is where we started to see the inflection point that's brought us to the vertical growth that we're at today and so that's been really exciting I mean some of these things you can go in the repo check for yourself, right? And gut check me on some of these things in terms of number of people, number of commits and all this sort of thing. And yeah, so it's been quite a ride. You mentioned the local versus the remote version of these coding agents.

49:21And it sounds like you have a lot of conviction that the right long-term direction is not the local versions. It's remote and in the cloud.

49:29Michael Bolin:Why is that? Well, I think that the people who actually for whom it is sticky now, and I think there'll be more of it, is that you imagine, if you imagine you want to just automatically, you know, set for the agent at every like GitHub issue or linear tile thing that comes in or anything like that. I mean, obviously, you know, there's, there's costs with those things, but, or you can, it could be abused, but like, you know, let's say for an internal private view, right. That you, yeah, want to be able to just have it be a piece in any sort of automation pipeline, right? And so you can't have all that happening on your laptop.

50:06Michael Bolin:And so I guess, I mean, as an individual, I see maybe we'll still personally spend more time with the local agent, but in terms of compute time of agents doing work, right? I think getting that set up in the cloud, I think a little bit, it's like a little more to get set up right now, but once you have it, it's quite nice. I see. Okay. So you're not seeing that product, the local one, will change. You're saying that across the industry, the compute that goes towards agents, majority will be in the cloud. Yeah. I mean, even in, I think when the Reus Code extension first came out, one of the things was the ability to take the conversation that you were having and hit a button, have it transfer to the cloud if you were set up to do it.

50:51Michael Bolin:Right. And so I think that we'll continue to see that where you're working on something, you want to throw it over here. you're like bring it back you know when it's done your codex usage is uh 5x since the beginning of the year and over a million people are using it now i'm curious has your ai workflow changed a lot since you've uh you know started to use this newer version of codex yeah it has it has um i'm a much bigger user of the app now maybe than i thought i would be um i uh yeah like for a while I was very strongly in like our VS code extension. Like I need, I want that sidebar. I want all the code there next to me.

51:33Michael Bolin:I feel like these things should be, you know, together. I'm not a person who doesn't look at the code. I'm not a, like I don't, for projects that are like true prototypes that are throwaway, like I will actually not look at the code. It's very freeing. I understand. I'm so excited about it. But for the code that goes into codex itself, I'm like, no, I still need to look at this, right? This is pretty important. This is like, what's gonna, you know, affect everybody else. Um, but you know, you start to get a sense of, okay, I like, I'm confident the model's going to be able to do this, you know, this, this change and this sort of thing.

52:05Michael Bolin:Like I, you know, I don't need to babysit it. I'm going to just like write a lot in the front. And so, you know, I have like four or five clones of the Codex repo of my machine and I, I have enjoyed in the Codex app, um, like the multitasking, I just is, it's just a lot easier because now you're just kind of hanging out in one window. um and you know i try and it's not just a game exactly but it's like you know it's like how much throughput can i get right in terms of how many balls can i juggle in the air sometimes it feels like um it you know it can it can feel like a little hectic when you're really trying to context but at the same time you're like i i'm getting a lot more done you know and uh and that you know, I almost feel a little bad writing code by hand sometimes because you're like, I could have if I had asked this in the right way, I, you know, it's like, it's like when you started, it was like, Oh, I'm just gonna change these three lines.

53:01Michael Bolin:And then like, you know, 30 minutes later, you're like, Oh, I kind of, you know, that, you know, we all we all like to type still, I think. And that's what I think. What percent of your code would you say is human written versus the model generates it these days? Oh, man, it's models that would be like 80 to 90%. I mean, yeah, I mean, like, especially like debugging a test or like the CI thing is bad. Like I'm like, I'm like, hey, you know, you know how to write, you know, print debug, whatever. Like, and that's great. That's really freeing. Digging into the problems that are suitable for LLMs and the ones that are not like what, what do you need to see where you think, okay, I need to go in and, you know, write it myself.

53:43What's that 10 %? And also what's in that 80, 90%. Yeah, I mean, that's a good one.

53:49Michael Bolin:I think about that. You know, because every time I sit down, I'm like, like, should I write this? I mean, the answer is almost always no. There's things that are like lower level and someone, you know, so Codex, the harness, the part that I spend my time on is in Rust. And that means that, you know, we can do and we do do like operating system specific things in that code base. and actually and I spend a lot of time on on the sandboxing right so the thing that really upholds like you know the security integrity of what of what we're doing and the model can't go outside you know the bounds that you set you know I I do more of that by hand because I need to make sure really sure that that's all correct right that our test coverage is is good or sometimes I'll seed it and then once I've got the groundwork and I have the pieces that I had a lot of feelings about and then could fill in the rest and that sort of thing.

54:50Michael Bolin:But a lot of refactors and things like that. Actually, the thing I like a lot right now is I'll have to build up a big PR that does too many things but I know it does too many things. So they're like, okay, please split this up into reviewable sized commits, things like that. I think about how much time I probably spent on that sort of stuff before. And now I can... Ray Alligate frees you out. What about code review? Is there... What percent of lines of code do you... And at OpenAI generally, are people reviewing manually? Or are there agents that are like... Imagine you write a test or something.

55:30I don't really need to review that. Yeah.

55:32Michael Bolin:I would say I like the approach where you know the the agent should do you know multiple rounds of review or whatever until it's confident and that it that is like i'm you know worth a human's time of looking at it um but we still do look at it um like ultimately before it goes in i would say generally speaking um i think there's you know we have like our agent's empty file like everybody else does there's still sometimes you find like a gap in knowledge that needs like that some context that needs to get added back into there or it's um yeah just just things that we haven't yet memorialized that that the agentness but like as a human i still happen to know right um and things like that so we do catch things um but you know i'll say actually now that people are uh also using ai to write their like pull request summaries um our summaries across the team notes they're getting way better so now at least also when i'm going into review right it's been reviewed by codex there's a summary that's like that we actually make sure it has like the why and the what of why you know the reason behind the pr so that is certainly i think helping like get through these prs faster which is good because there was a lot more yeah review to do that's i mean that's amazing i feel like 50 of diffs have like you know almost blank the test plan so you know uh i don't know art build or whatever you thought.

56:57Oh, I know, man. I want to talk about the Codex CLI that's open source. Why is it open source?

57:08Michael Bolin:You know, for something that's that critical to like what's going on on your machine, right? Like that's one of the aspects of open source. I'm not, you know, the most zealous about this sort of thing, but I sympathize with it, this idea that like, hey, you're going to put this thing on my machine. I care about what it's doing. And I think in this domain in particular, it's really important that people can look at it and have an idea of what it's doing. Because people have a lot of questions about AI agents and that sort of thing. And so I think this area, this domain, I think it's actually really important.

57:51Michael Bolin:And also we have gotten a lot of, I think, great contributions and bug reports and things like that that we would have missed out on. And then also, I think it's just sharing with the world with how this is done. So we do it through code, right? I've put out one blog post about how the agent loop works. There is a plan to do more of that. I'm actually excited to. It's just time. Is this the limiting factor? It's a resire. It's not really not. Yeah, it was funny because I had two candidates who came through and one's like, he's like, hey, I wrote it, right? I was like, no, no, I wrote it. And another person came in.

58:31Michael Bolin:He's like, oh, you can tell that, you know, that you did not outsource the, okay, good. Right. I was like, oh, thank you. You know? So, yeah. You mentioned the blog post. How does Codex find what is available to it in its environment? When I'm running these things, I'll see, it's kind of amazing. It's thinking to itself and like discovering all these things in my terminal. So, yeah, how does that typically work? Yeah, I mean, there's a few. I mean, there's obviously what is, you know, in Codex's base training, like it loves to use RIPGREP. It uses RIPGREP very well to find all sorts of things.

59:05Michael Bolin:And then there's, you know, if you have your agents MD file and you're saying, hey, in this repo, like these tools are really important, right? You should use these, right? Or the readmes or whatever. or obviously if you use mcp and you associate these mcp servers with your um you know where you're working on right that that that um injects the set of tool definitions like at the start of the the conversation right so then uh i kind of yeah so that's that's not even like discovery on codex's part right it's kind of just like put front and center there i see so some of it's the harness explicitly throwing that into the context and then there is a big chunk though where the model is just doing all the heavy lifting of finding things is all right yeah kind of like reflecting over your career the breadth and depth of your work if we look oh across all of it i mean it's it's insane you're javascript front end then you have all these dev tools you know build fuzzy file search, you know, virtual.

1:00:05Now you're working on codex. I'm sure you had to continue your engineering education to kind of get through all these projects. What are the top technical books that have helped you educate yourself?

1:00:20Michael Bolin:Yeah. I mean, you know, so one was this book on operating systems. It's like a thousand pages or something like this. It's the Addison Wesley book, I'm trying to remember the author. But it was funny because I was, when I was working on the virtual file system, so I had actually gotten to that point in my career without ever writing C. Like undergrad, like it was like more theoretical, like we just never had to touch it. And then yet I'm like working on a virtual file system project. And that was like very low operating system. That's why I was, you know, joke that I was kind of the worst engineer of the project.

1:00:53Michael Bolin:And, you know, and someone said something and I realized, I was like, I don't know what they're talking about. This is kind of embarrassing. And I was just like, you know, what book do I have to read? And I think my manager, Aaron Kushner at the time was like, he's like, well, there's this thousand page book. And I was like, done. So I bought it, read the thing cover to cover. I like took it to Hawaii with me. I took it everywhere until I finished it. It's kind of amazing and sad or weird, like how, how far many of us can get like in software engineering without really having a clue how computers work like there's just so many of those levels of abstraction um but you know on one hand it's very freeing and then on the other hand it's little bananas and i think that um what i would say to people right now is um is you know actively trying to go like deeper through the layers and understand these things um and uh like because many times i saw other people do it and now I can do it is that there are problems some other people could solve that I couldn't solve because I just didn't know that like there was this cruft between these two layers and if you got rid of it right you get like a 10x improvement or something like that right if you if you're just operating so high up you don't really know you know what you can what you can break down you know so that and so in terms of books like I've enjoyed the like the O 'Reilly the Rust books I you know big kind of rust I think that they're also like well-written and thorough.

1:02:21Michael Bolin:And then honestly, another thing I've been telling people that's not in the book category, but probably more fun. And I learned a lot actually are the CTFs capture the flag, like security type competitions. And just, you know, it helps with like kind of like adversarial mindset. It helps with, I think they're usually just like, it's like a, it's like a decathlon like a computer decathlon i feel like right because like like there'll be multiple challenges and maybe this one you need to like understand assembly and this one you need to understand like what someone's like janky php admin page is doing and all that sort of thing and it and it just forces this um breadth on you uh in a way that's kind of hard to to generate otherwise and it's also you know just more fun because it's like a game and that sort of thing.

1:03:12Can you give some context on what CTF is?

1:03:15Michael Bolin:Yeah, so it's, it's, it can be a number of things, but it's usually a competition, usually in like the infosec, the security domain. And there'll be a set of, there's like the Jeopardy style one, which is like, there's a bunch of challenges. And like, you know, designed, like crafted ahead of time, and they all have point values associated with them. And so, you know, there's usually some fixed around a time, and it It could be individual. It could be with a team. And you're trying to solve these things. And there's a flag. There's always like a secret, you know, piece of text in there that usually has some format.

1:03:51Michael Bolin:And the way is that you basically, if you can discover the secret piece of text, that means that you have the challenges set up that you would only have gotten that if you had figured out, you know, reverse engineered or whatever you were supposed to do. And then you get submit that piece of text. And that's your, you know, token to demonstrate that you solved the challenge. And so it could be like a race to like solve everything first or get the most points in a certain amount of time or different things like that. So it's like it's kind of like an escape room, but in your terminal. Yes. You have only computers to figure everything out.

1:04:19Yes. I see. And your recommendation is people who want to become better engineers, they should invest in doing some of the CTF because they make you solve problems that make you a better engineer.

1:04:31Michael Bolin:Yeah. And, you know, and then you, you know, also develop skills and things that you just wouldn't have. You know, if you're just like, you know, writing React every day, like you're probably not going to like open up GDB and like reverse a, you know, a tic-tac-toe game. But like I did that because of, you know, a CTF challenge was to do, you know, exactly that. Right. And but then it's like, but then I learned how to use GDB. And like, and then you start, you know, when you're faced with other problems, you're like, oh, my just toolkit of ways to solve things is just much broader now. A lot of people, especially earlier in their career, they see the power of Codex doing everything for them in the terminal.

1:05:10And I just imagine them saying, oh, I don't need to learn GDB because Codex knows it. So what would your advice be given the landscape of these AI tools for people thinking about their engineering education?

1:05:24Michael Bolin:Yeah, no, that's I mean, I think everyone's like struggling to answer that question right now. I do think about it a lot. I don't have a great answer. I kind of personally come back to my thing about like, I still think like trying to forcefully kind of go through the levels of abstraction and just understand, you know, more like at a deeper level, how things work is going to be important. I mean, I think this will change. I'm sure this will change over time. But right now it's still kind of like, you know, the questions that you asked the agent is going to affect the quality of the thing that you get out.

1:06:03Michael Bolin:Right. So if you're not asking the right questions, you're not maybe going to get the best engineering solution out the other side. You know, as things go on, perhaps that will also be, you know, another layer that's removed. I mean, I think, you know, that's certainly where we're going. I don't know what time frame we'll get there. Things do seem to be happening faster than we expect. But, yeah, I think in general, learning how to ask what the right question is. And I haven't, you know, for myself even, like, totally pinned down what that means for someone who's, like, starting out new, right?

1:06:35Michael Bolin:I, you know, unfortunately have experience to fall back on. Or, like, that's where that taste or that intuition of what to ask has developed. But, you know, if you're starting out, I'm still not sure yet. And it's also hard to say because we don't really know 100 % where everything's going. When you reflect over your career, the expectations at these really high levels is kind of crazy. I mean, it's like already for most people, this E7 or senior staff level is this unattainable level of impact. So someone who gets promoted to that level is thinking, I got to really work super hard now because the bar is like up here.

1:07:17and then you went two levels past that and so i i'm curious like in the day-to-day now what what are your thoughts on these like crazy expectations is it stressful for you

1:07:29Michael Bolin:i mean it never it was never not stressful i think um for me because i really i think also because i sat in calibrations right and i um you know so for a bunch of other people you know you You talk about their level, you talk about their impact, and you want to be really fair and have this integrity around, well, this level means this thing, this is the impact. Like, that's what it meets all is. That's what it exceeds is. And then knowing that, like, then you're, you know, playing it out in your mind, like, well, someone's sitting in my calibration and talking about this thing, and they want to be fair, right?

1:08:01Michael Bolin:And it is a little scary. Like, you know, you get to E8, and you're like, oh, you know, it's a D1 or a director. And you're like, how do I have as much impact as a person who has maybe, like, over 100 people in their org? And so that's hard. A lot of people do it. Again, a lot of people who are an IC who do it, do it more by a different form of people management, right? Where they're trying to, you know, write the right doc and get the people aligned and do that sort of thing. And the reason that they do it and are not a director is that oftentimes the people, I've talked to people who are in this, like they're like, well, I have this technical credibility or because I built this thing.

1:08:39Michael Bolin:Like when I go to the team and talk to them, it lands differently than if like a engineering director does. Some version of that, right, happens a lot. And so then, you know, when you when you say you're influencing, you know, 50 to 100 people as a senior IC, then you're like, oh, OK, that's like, you know, D1 impact. um you know whereas as if you're a coder i see um uh you know being really thoughtful about the projects that you pick and it can't just be like this is fun for me type of project i mean you know if you actually care about you know not getting fired or getting or you're getting like sam beats all or or better um and even when i would start a project certainly like the later i went on i would think a lot about like okay like i maybe there's this feature i just want to write it because it's fun and i'd be like oh no i should let somebody else do that and think about okay i mean it's kind of like with with codex now like what code should i personally write to maximize my impact versus if someone else could do it about as well as me let's say 80 is good i should probably let you know have someone else do that um and but but even then you know if you're leading a five-person project is that still, you know, still getting E8, E9 impact still hard.

1:10:01Michael Bolin:Um, so really finding that, that, uh, project that's a force multiplier. So I, you know, like, like, uh, like the virtual file system was a really great project because that was gonna, you know, we knew that down the road, like this was really gonna unlock so many things, prevent us from, from being completely blocked. Right. Um, actually another big part of it is, um, and And, you know, one of my managers talked to me about it is, you know, not actually recognizing senior managers enough who are the person who pairs the senior IC with the right project. And because some senior engineers are amazing, you know, fixers slash coders, but they're not the idea comer-uppers with.

1:10:49Michael Bolin:and they're the, but they're the person you want for like, you know, very difficult technical projects. I don't want to like out anybody here, but I have people in mind. And, but, but, you know, a lot of times it's the manager who realizes like, oh, this project needs this person. Right. And like that person would have never realized it themselves. One of your old colleagues, Adam Earth, I think I asked him, you know, what are some other engineers that you admire and why? And he mentioned your name, of course. And specifically, he talked about your ability to start projects was really good. And obviously, I mean, look at all these projects, right?

1:11:23I mean, a lot of them, you created them out of nowhere. Like it was just, you had an idea and you went off and build a prototype. You came back and you were very convincing that it was a better solution. Do you have any advice for engineers who, they have a problem and a solution and they want to like build a project from scratch?

1:11:41Michael Bolin:I mean, I think, I don't know, a lot of good projects, I think come from being a little bit dissatisfied about, about something. Right. And I think, um, you know, it's funny, like sometimes I, uh, for better or for worse, I was just so charging ahead and just like building the thing that like not really thinking about, um, like what the best way to do it was. So actually a really funny example is, um, Google calendar goes, they're going back really far. I was like, I want weather. I want to have weather, little icons showing the weather in Google calendar. And I was like, I'm going to do it, you know, and I would mostly been JavaScript, especially and not done any of the back end stuff.

1:12:18Michael Bolin:And I just like charged through and I cobbled things together and, you know, made it happen. And then my tech lead was like, wait, how are you? How are you storing that information about like the weather and stuff? And I was like, oh, I just like, you know, threw a blob of like XML and everything. He's like, we should have talked about protocol buffers. I was like, you know, like binary formats to save bytes that I like, I was just so, you know, set on weather of all things that like I never bothered to ask anybody if there was like a better way to do what I was doing. I was like, it's done. So, I mean, I guess that's a unique skill of yours is the digging into the dissatisfaction and solving your own problems.

1:13:00Seems like almost every single one of these projects is I want my I want something to happen. This shouldn't be this way.

1:13:07Michael Bolin:And then you went and you solved it. Yeah. Yeah. But I think another unique thing that I see is a lot of people, they get to work and they, I don't know, they're in their dev environment. I'm sure there were many other people who saw with the buck story, for instance, they go, oh, wow, this build is really slow. Oh, well, like, I guess I'm going to go to the micro kitchen and get an earring call that's builds. What gave you the confidence to know that you could make it so much better? I mean that one was you know I have to credit like having been at Google I never worked on the Blaze team I never knew how that like worked ever touched the code or anything but I was like I know that there's a thing out there that has this shape and is a lot better than this thing so that's an existence proof whether I could do it or not like TBD right but I think that was helpful on that one you know yeah I mean there's a lot of I guess prior art that gives me you know, confidence and things.

1:14:03Michael Bolin:And I, you know, I usually generally identify myself as a coding machine, I guess with Codex now everyone's a coding machine, but that, you know, I felt that I always had confidence, like, you know, build a prototype quickly, right. And at least answer that question or like the basic, my test, my basic hypothesis, like spiritually, should there be a way forward to this thing that I think should exist? And, you know, generally like, you know, if you are determined, you'd find a way. Yeah. I've noticed that pattern. A lot of people go go to big companies, they see this world-class infrastructure, and then they go to the other company, oh, you don't have this, you don't have this, and they build their, you know, those new versions of it.

1:14:41I've read a lot of your writing at this point, and it is so clear. It's some of the best examples of good technical writing. What advice would you have for engineers who want to write better?

1:14:52Michael Bolin:I mean, I think a lot of it is, well, I think reading other good writing is a certainly a good start right it started maybe you know consciously or subconsciously started to pick up patterns of like what what is out there i think um you know really a high level thinking about like what what what is it i'm trying to convey what would someone actually really want to know um and and outlining a lot up front right is i think a lot of people give this feedback but it's it's impressive how how really important that is and just being like does this set of things like linearly follow i think is a big thing and and and then asking yourself oh i went from this point to this point was that too big of a jump is there something that like somebody actually like would have reasonably missed and i think uh personally i feel like that's i have a reasonable feel for like what where that gap would arise and like and then if you can kind of anticipate that and magically put in the example that like someone would have that someone needed to make that jump.

1:15:53Michael Bolin:I think like that, at least for technical writing is like, is a big deal. You have that career note that I love. And at the beginning of it, you lay out this three-step plan for impact. Can you explain that three-step plan? I feel like that's a good algorithm for people to use in their careers. Step one is figure out what you really like to do. Right. And, you I mentioned like, it's good to broaden that, but it's also good to, to be honest with yourself. So you don't know. And like the, the quote, the hero quest that I went on that, that didn't pan out because I was working on stuff that I didn't really, you know, truly love.

1:16:31Michael Bolin:Um, and then two, step two was figuring out what your employer is, what's really invaluable to them. Right. And as I talked about it, Google, I didn't do a good job of that. I did stuff that I was really excited about, but it wasn't, you know, it wasn't, you know, AdWords for Google or anything like that. Yeah. And then step three is like, find that intersection and then just really lean into that. And, you know, the more that you can do that, I think the more successful, you know, you're going to be. And the challenge is sometimes it's not always there, right? And maybe you have to go, you know, find somewhere else to make that happen.

1:17:06Okay. And then, yeah, last question for you is, if you go back to yourself at the beginning of your career knowing everything you know now

1:17:12Michael Bolin:what advice would you give yourself i think i should have been open to learning more things sooner and and i i you know to be a little gentle to myself and i think other people are in that situation is that you know there's so much to learn when you're starting and then whatever your first programming language is i think it's funny i think everyone has a soft spot for they'll make excuses for it for like ever like oh no it's a totally good language and i think it's because it's like the first thing that enables you to do a thing you know do anything right and and then it's like oh okay i can finally do something it's such a relief and um and but it's also like a hazard because like then you kind of want to hold on to that thing because like you're finally productive and now you're like oh it took so long to get to this foothold i don't know how long it's going to take to get to the next foothold so i think like in you know my particular case i probably I did maybe went too deep with JavaScript.

1:18:05Michael Bolin:And like I said, it took a long time before I wrote NEC. And I think, you know, if I had been a little bit more curious and a little more flexible in terms of what, like, types of projects I was willing to take on or things I was willing to learn, you know, it came eventually. But I think that is probably the biggest thing that maybe could have made a shift for me earlier. Yeah. I gather from your story there was a point with the Xcode where you said you hated Objective-C and then you were coming up with like ways to compile the Objective-C or Java into Objective-C or something like that maybe.

1:18:42Michael Bolin:And then, you know, also talking about the C++ for Miles, it was like, it seemed like a very concerted, okay, I'm going to learn this as opposed to like just kind of being open to it. So, yeah, that makes sense. Well, maybe with Codex in the future, it'll be less of a hurdle for people. You could just kind of say, hey, I know JavaScript. Write this in Rust or whatever. No, it's true. Opens a lot of doors for sure. Awesome. Well, thank you so much for your time. I really appreciate it. All right. Thank you, Ryan. Thank you for listening to the podcast. It's a passion project of mine that I really enjoyed building.

1:19:15Another passion project that I've been working on kind of in secret is building an ergonomic keyboard that I wish existed. And I finally have a prototype. So I'd love to show you what we've built. It's ultra low profile and ergonomic, and I couldn't find anything like it on the market. So that's why we built it. I'll put a link to the keyboard in the description. You can take a look and learn more about the project there. We could definitely use your support. Also, if you have any feedback for me about the show, I'd love to hear it. Comments on YouTube have led to guests coming on like Ilya Grigorik and David Fowler.

1:19:48I wasn't aware of them until someone dropped a comment. Also, feedback in the comments helped me learn to reduce the number of cliffhangers in the intros. So your comments definitely make a difference. Please keep letting me know what you'd like to see more of in the show and I'll see you in the next episode.