What is a Principal Engineer at Amazon? With Steve Huynh

26:22Once that sort of breaking point occurs, then you start to try to figure out how you can sort of break things up. But starting with a microservice architecture, especially when you're small, like what a waste of time and energy. Totally. So you were a principal engineer on Amazon. And apparently I learned that most companies, they have different levels. And again, this principal engineer, some companies have like staff level, but it's usually like entry level, mid level, senior. And then you have staff or in the case of Amazon, it's principal. I've learned that Amazon's principal level is both really hard to get into compared to a lot of other companies.

26:59And it's a it's pretty special in some ways. So we'll talk about that. But can you tell me, like, how how is the career kind of development? Because most people imagine like, oh, it should be pretty straightforward. I spend like, I don't know, two years as a junior, two years as a mid roughly and two years as a senior. Then I get to principal. How does it actually work at Amazon? I think it's linear up until you hit principal. Right. So you join, you're a junior developer, you get promoted to mid. At mid, you're starting to influence the team, but then you get to senior. And so now your expected impact is at the team level.

27:32And then there's this jump that you get to principal. And principal is L6? Principal is L7. L7, yes. And so I think you really have to start with why is that jump so big? Because I think at pretty much any other company, it's just a linear progression. Like there's nothing necessarily special about staff. You know, you can just sort of go to that level of senior staff and then principal. But for some reason, Amazon decided that they weren't going to have a staff level. And so, and I think they sort of like couched it around like having high standards. Basically, to get from senior to principal, you have to do like two and a half level jump.

28:14From L6 to L7. Yes. Technically, it sounds like one level, but at some other companies, this might be like, you know, L8, L9 or L8 and a half. Yeah. And, you know, so the hand wavy argument is like, hey, we have high standards and like, you know, it means something to get to that level. It's like, fine. But I noticed that some of the best engineers that I'd ever worked with were having such problems getting to principal engineer that they ended up moving to Facebook or to Meta or to all these other places where the progression was just sane. Now they're - So they got a staff or a senior staff.

28:47Now they're senior staff and principal and distinguished engineer at other companies. And so because we had high standards, we actually had this brain drain. And it wasn't a brain drain at lower levels. It was the brain drain at sort of like the higher levels. And it's just an example of something where it's just like, why did you do that to yourself? And so that's the context for being a principal at Amazon. It's safe to say it's wicked hard to get in front of you. Right. So, you know, I am colleagues with Ethan Evans. And so we talk about what's the hardest promotion at Amazon. And, you know, I had made the argument that it was, you know, it was senior engineer to principal.

29:29And he's like, yeah, that's hard. Actually, the hardest one, Steve, is VP to senior VP because there's only eight spots or 10 spots for that and maybe 300 VPs that are all trying to get this. That's more of a supply and demand thing. I will say that at Amazon, there is gigantic demand for principal engineers. And so there are roles that have been open for years. I think something on the order of like 13 months or 17 months or something like that to get an external hire to join as a principal engineer. But that metric is only calculated when the role is filled. And so probably, you know, there are hundreds of principal engineer openings at Amazon and there are thousands of senior engineers who desperately want to get there.

30:15That would love to be putting in the work, you know. And so there's this sort of like, there's this tension, right? And I don't think you see that at the lower levels. I don't think that that's happening at senior or mid or junior. And so like that incongruity, I think, is super interesting. But once you do get to principal engineering, one thing that I've never heard any other company have is there is apparently a principal engineering community, which is I've heard, again, from other people that it's tightly knit. It's actually special. It's actually just a really nice organization. Can you talk about that?

30:47So like, you know, once you once you got in there somehow, I don't know, was it bloods, foot and tears at promotion? There is a community. I think it's actually really great. my own history, you know, I went from support engineer to senior engineer in like four years at Amazon. But then from senior to principal, it took me eight years. And I got promoted in Q1 of 2020. Turns out to be a consequential like year for in the industry for the world. That was forceful remote work. Yeah. And so, you know, I got promoted and everybody's like, you know, congratulations. They used to have like a principal engineer offsite where they just flew everybody into Seattle or nearby and then to sort of like, you know, mingle and to talk to other folks.

31:32That stopped during the pandemic. And then, you know, by the time the pandemic restrictions started leaving, the population of principal engineers had essentially doubled. That's still to say, like, there are still hundreds and hundreds of openings for principal engineer. But then the, you know, the sort of like offsite community shifted over to the senior principals that I didn't have access to. But, you know, at the moment, the manifestation of the principal engineering community is essentially through the Slack channel, which is absolutely awesome. And then we had principal offsites for like our local organization.

32:09So like Amazon Music, Prime Video, Twitch, that sort of thing. Those meetups were amazing. So the reason they were is because of this high standard that Amazon had created. And so what it meant is that everybody that was able to achieve that overly high standard, there's something exceptional about them. Um, there's, there's, you know, um, they're super deep in a particular technology or they were associated with, you know, uh, the growth of a really large line of business, either within Amazon or, or externally, they were essentially leaders within the industry. And you could just literally, you could just scoop out five people and then put them into a room.

32:52And the conversation is just, it's just amazing. Right. And I would sort of be like, I don't even belong here. Like, look at this guy. You know, he wrote a book on a particular topic. And this guy, you know, he was a luminary in a particular field. And then this person just like is an amazing code machine and can just write an entire application over a weekend. And then you're like, what am I doing here? I do wonder if that community might be coming back now. I know you've left, but now Amazon is now in person because it sounds like a lot of the benefit was the in-person part as well. Because this is what I never heard.

33:31Even before the pandemic, I didn't hear other companies say, for example, Uber. I've heard that the senior staff engineers do get together every now and then, but it was very like roots. So it was bottoms up. But my understanding at Amazon actually invested not just, you know, some principal engineers saying, hey, let's get together, but also just kind of, you know, like making sure that that group really had something. Like I think it's smart. I think more companies should do it, but I'm just not seeing it. The investment was also in terms of headcount. So there are program managers and like product managers, essentially, that are, you know, bringing the folks together.

34:12Oh, awesome. There's a wonderful series. It's called the Principles of Amazon series where, you know, principal engineers will just, you know, they'll do a presentation and it's recorded. That's been happening for, you know, 20 years. And, you know, we record everything that's there. But it takes work to actually. That's an internal series. Yeah. And is that open to like everyone at Amazon or it's for the principals? Oh, it's open for everybody at Amazon to consume. To consume, yeah. And then, you know, there might be some senior engineers and stuff like that that would make a presentation. That's part of their promotion packet.

34:45It was to be able to make an Amazon-wide presentation on a particular thing. My point was, though, that that stuff doesn't just happen on its own. Like you have to, like you need a program manager or multiple folks to sort of like herd the cats and to like schedule the off sites and to make sure that the, you know, the Slack channel doesn't go off the rails. Right. And it's still useful. And it's just not going to happen like grassroots with just like throwing a bunch of people into a room. This episode is brought to you by Augment Code. You're a professional software engineer. Vibes will not cut it.

35:20Augment Code is the AI assistant built for real engineering teams. It ingests your entire repo, millions of lines, tens of thousands of files. So every suggestion lands in context and keeps you in flow. With Augment's new remote agent, queue apparel tasks like bug fixes, features and refactors, close your laptop, and return to ready-for-review poll requests. Where other tools stall, Augment Code sprints. Augment Code never trains or sells your code, so your team's intellectual property stays yours. And you don't have to switch tooling. Keep using VS Code, JetBrains, Android Studio, or even Vim.

35:54Don't hire an AI for vibes. Get the agent that knows you and your code base best. Start your 14-day free trial at augmentcode.com slash pragmatic. I think, you know, these are the things, I mean, we're now exposing a few of these things here and there but some of these companies like you know amazon is a great example where there's more to the eye than what meets the surface so like once you're inside amazon for example you now as an engineer even if not a principal engineer you now have access to the whole you know 20 years of principal presentations like when i joined uber i was amazed at how we had the rfcs available like i could read all historic ones so i think there is and every company has its own of course once you're in there you have access to this like knowledge base which it will just never be published it cannot because it has you know business sensitive things etc so i think as an engineer like you can just really just like like be a sponge when you join especially one of the companies that is known to be a bit more open internally even if amazon i think a really interesting one because externally it's very closed is my sense they're very careful about what they share for example the postmortems for aws is very few are published externally but internally they're all there as i understand there as an you can access, you can learn from them.

37:05Like really cool real world learnings. Absolutely. You know, it is an open place internally and we are so selective about what we, I say we as though I still work there, what they publish externally. And, you know, the postmortems, we call them COEs. It's a correction of error. It's, you know, it's this idea that, you know, you have like holes in Swiss cheese and you have like a failure requires that there's a hole across layers. That's the best reading. Like I would just subscribe to the email list where they were published internally. So you have this like stream of like of disasters that are going on within the company.

37:46And you just, you know, you grab some popcorn and you pop open one of these COEs and you learn so much from that. And I think that that's part of the secret sauce. The idea, and I don't know if it's like this for 100 % of them, is that it's a blameless culture sort of thing. And so to really screw up requires that multiple people drop the ball. And you learn so much from that sort of stuff. You know, the brownouts, you know, these lessons that you would learn from, you know, trying to recover from really large dependencies, those things are immortalized inside some of these COEs. So there's some very famous outages that happened within Amazon.

38:27And, you know, there were an egg on our face. And we really, really learned those lessons through those postmortems. They're absolutely wonderful. As a principal engineer, so far we kind of glamorized roles, saying, you know, it is hard to get into. But once you're there, you have the community, you do this really impactful work. But one of the principal engineers at Amazon who's still there called Bavik Kotari, he collected some things that are maybe not as glamorous or more challenging about principal engineering. He had five of these things, or five or six. I just want to go through with you and your take on this.

39:00So first he wrote, there is this paradox of belonging that you're part of all teams, yet you're part of none. What does that mean? Yeah, no, so Bavik was actually a peer of mine. We worked in Prime Video together. Oh, awesome. So he's an awesome dude. Yeah, there are all of these paradoxes. And this paradox of belonging is a really interesting one. you work for the organization. You're working across teams. So as a senior engineer, you're embedded on a team. And you own the team's architecture, the operations, the software development lifecycle, and the design. But when you get to that next level where you're working across teams, you kind of operate in this weird layer where you're not on pager duty for a particular team, you have visibility across all of these teams that are there.

39:58You're helping to guide and make decisions, but you're literally not on the ground floor anymore. And so, you know, when you work with a particular team, you know, you might call the senior engineers or the mid-level engineers in and be like, hey, let's whiteboard some stuff. Like, let's try to figure out what's going on. You're not on the team. You're kind of this like advisor that's sort of coming in, right but then you know maybe a director or a vp would call you in and say like hey what do i own like what's going on explain to me this outage or tell me why we can't build this thing and then you're you're trying to whiteboard the architecture and the system and you're trying to say like hey you know this is what's going on on the ground floor but you weren't you know you weren't part of that team so you're just sort of operating in this this sort of strata where you know you don't really belong on a team.

40:48You know, I'm a, I'm an immigrant, I think you are as well. And, you know, my parents came from, from Asia. I'm not Asian, right? So when I go back to Asia, I'm definitely from the U S and then growing up in this country is just like, you know, I'm, I'm, you know, not quite an American, right? And so you, you sort of operate in this sort of, you know, area in the gaps where your identity is really defined by not being squarely in one of these predefined categories. And so it's very similar to that as a principal engineer. You're not on the ground floor. You're not checking in. You will check in code, but you're not necessarily part of that team embedded on that team.

41:28And even if you are for a short time, it's usually a short time. And like tomorrow, the director will call you up and say like, hey, Steve, we need you on this other team. They're in trouble. Move over. Yeah, and you parachute in and then they're like, oh, who's this guy? And then your director is like, what's going on? What happened during this outage? Why is the press writing about us? And then you're like, well, here's what's happening on the ground. But you're not really embedded on that team. Which leads us to the next paradox that Bavik said. He lists a few of the paradox, which is a freedom of responsibility.

42:02And he writes that you enjoy significant autonomy in being able to choose what you work on. However, there's an implicit expectation and accountability for resounding impact. Yeah. So, you know, I reported to a VP right before I left the company. So they were your manager, basically. Yeah, my manager was a VP. Oh, wow. Wow, that's, I don't hear many companies having engineers report into VPs. Yeah. That doesn't seem very standard. You know, and so the org that he owned, you know, I considered myself the tech advisor for that organization. It was about 450 people, 450 software developers. And what did our one-on-ones consist of, right?

42:45Like when I would have our one-on-one, it wasn't like, hey, here's, you know, he didn't assign me work. he wasn't like hey i need you to build this thing i need you to design this thing the context that he said was basically like here's a direction right that you need to go and the way that you can achieve that type of impact was up to me right so he might say something like hey availability is so important for you know live sports we just signed you know billion dollar contracts with these sports leagues. And so we need to increase our availability posture. And then I would be like, okay. And then I would go away and it would come back and I would be like, you know, here's what I'm working on.

43:32Right. Like that type of dynamic, I don't, does not exist at the senior engineer below level where you're basically telling your boss what's happening. I was about to say that when you said my, my manager one-on-ones, he didn't tell me what to do. I'm like, most engineers would be like, sign me up. Like, I don't want, you know, we all hate micromanagement. But now when you're telling me, like, he would say like, oh, so we just signed a billion dollar contract. Availability is important. And then stops talking. I'm like, that sounds uncomfortable. And basically, like, you're kind of expected a little bit to like, understand what he's expecting, even though he doesn't know.

44:06And then, and I'm assuming, you know, there's two ways of going, right? You go back on the next one-on-one and you say something and he was like, like, Steve, like, you're a principal engineer. This is not what I expect of you and you don't want that. Whereas this, you know, if you bring back the right things, it sounds like you really need to up level and like understanding how these people think. Absolutely. And so he's, you know, he's accountable to his boss as well. And, you know, don't get me wrong. I didn't, you know, I had a, I owned aspects of availability. You know, there's a multi thousand person organization at Prime Video doing this stuff, but we own the live sports aspect of this.

44:42And, you know, there are playback teams, there are, you know, recommendation teams. There are so many different teams that are there that had to really step up and make sure that availability was good. But he would say something like, hey, what is our availability posture for certain aspects? And I would have to go and figure it out. Yeah. Like, what are we measuring? What are we not measuring? There's a deadline for the start of a season where we're expecting millions and millions of concurrent to come in. What can we do between now and then, right? And then if we do write some software, like what is the highest leverage piece of software that we could create that would increase our availability posture?

45:24And so the way that I sort of describe it to people is you are assigned not a problem, not even a problem space, you're assigned a direction. You can solve the problem with code, you can solve the problem with system design and architecture, but you could also solve the problem say by, you know, I don't know, hey, maybe there's some off the shelf software we should purchase. Maybe there's a dev team that we should start to spin up right now whose job it is to do this particular thing. Maybe we've identified a piece of software and it's already been scoped that this team needs to go and build, but it's not a priority for them.

46:01Now we need to go and figure out like, you know, how we can get them to do it. Can we shuffle around resources, that sort of thing. And so the way I describe it is like, there's so many more things on the menu that you can use to solve the problem. And I don't think people recognize that. They think that it's just, oh, when you're a principal, like you just like code a lot and it's just really complicated. Or do more meetings, you know, that's what often happens. I mean, at the end of the day, like don't get me wrong, there's a ton of meetings that go on. Yeah, yeah, but this is, I think it's good to like shine light because I also feel like once, it sounds like a big change, but I also kind of feel if you get good at this, you might not really want to go back So, you know, having a manager is like, all right, here's a project we need to solve, like, you know, scope it out and what you can do.

46:45Right. Yeah. That's cool. And now the next challenge that Bobic said was this all sounds great, but there's apparently a bandwidth challenge. So it's it's become this like social resource where people just pull you into everything and you're breathing. Yeah. No, you know, I think I wish I had taken a screenshot, but, you know, I have my Outlook calendar. Right. So it's my schedule. My day looked like most people's week. so it looked like somebody had just like blew up a tetris factory like there was like i would have triple or quadruple booked on a monday all through the day you would have the manager calendar as an ic yeah and it's it's absolutely crazy because and you know for that large org that i was supporting everybody just added me as optional or or they might try to say like no you're actually required for all of these meetings but when you have you have a triple booked calendar and you're required for this stuff, you just learn that you're going to have to disappoint a lot of people.

47:42And so it's this sort of like, you know, this thing where it's like, it's almost easier to say no now that you're obscenely overbooked versus when you're a senior engineer, you're like, I don't have time to write code, but there's just barely enough time in between the cracks. Yeah. And so I think that it's almost like when your schedule breaks, that's when you are finally freed because you know that you can sort of say no to stuff. But ultimately, if I just went to all of the meetings that everybody said that I would have to go to, I would be a professional meeting attender and I would literally have no time to do the work.

48:17And then Bavic follows up on this next challenge, which is being truly present. And he writes, I think it's almost like, you know, he was sitting next to you. You find yourself physically present in one meeting while your mind is already racing against next three. You know, it's a really big challenge. You know, I pride myself on being a good communicator and being present. And when there are 20 things that are going on in the air or 100 things that are going on, it's just really, really difficult to say single-threaded. And what I ended up having to do is to sort of say, like, okay, I could do all of these things and they would be really impactful.

48:54But I just had to aggressively prioritize and say, you know, for the availability. I'm just looking at availability. There's all these other fires that are going on. which is disappointing because there's so many things that, you know, you could be focusing on. It's super difficult. And so, you know, I work with a lot of people to try to get them to the next level. And they say, Steve, I'm completely overwhelmed. There are like 20 things that are going on. And I tell them, like, do you think it gets easier when you get higher level? There's just going to be more and more things on your plate. Why wait until you burn out or you break?

49:29You can just start implementing these things now. So every high level tech I see, I know, and managers included, they have a wonderful system in order to like isolate signal and then cut out the noise. And if you don't have that, you literally won't survive. But it just at the at the principal level and above, it's just it's just amplified that much more. I'm getting a sense that a lot of the work as you do as a principal engineer, I mean, there's huge amounts of software engineering and you need to be just really good at building resilient systems, learning about new technologies. You know, for example, today, I'm assuming Kurov is a principal engineer at Amazon.

50:06They're expected to just know everything about LLMs, trade-offs, characteristics, et cetera, because they're anyway. But you also need to just become, do the skills that managers have, which is managing your time, changing contacts, figuring out how to get that focus time. Like, you know, contrary to popular belief, like managers actually need focus time. So, like, you know, I will always try to carve out some time. But you're now doing it while your title is not manager. But actually, it feels like you combine a manager, a lot of manager responsibilities and a lot of, you know, like experienced engineer.

50:40And boom, you get the principal engineer role. oh, the only upside is like, you don't need to do performance reviews for people. Congratulations, you've saved a little bit of that time. Well, actually, during performance review season, they pull the principal engineers in because if you're stack ranking people, okay, cool, well, we'll need to take a look at their performance in tech. So I reported to a VP, one of my peers was a director and he was basically like, hey, Steve, I would like you to show up to my performance review for my entire org of hundred something people. And I'm like, I can't do that for you and for everybody else.

51:13Okay, so now it makes sense why as a principal engineer, your compensation package will be similar to like, is it a senior engineering manager or something like that? Around that. Around that, but basically like the job has a lot of overlaps. Okay, the benefit is you're not the one delivering the performance reviews of the direct report, but you're doing almost everything else in terms of the effort I'm talking about. Okay, so having been a principal engineer for four years, what are the good things that you really liked about Amazon, specifically Amazon's principal engineer role? And what are some of the not so good or it could have been better things?

51:51I mean, the great parts are you get visibility that you just couldn't possibly have at the team level. Within a large organization like Prime Video or wherever you're at, there are many thousands of people that are working within that organization doing so many things, right? And typically the performance of these people is really high. There's so many different directions that are going on. And so to survive, you kind of have to look inward and you say, okay, well, here's my service boundary. Here's all the software I own. I'm going to own everything within the sphere of ownership. Because you've built this wall up, you tend not to be able to see like that broader picture.

52:29And so as a principal engineer, I think it's really awesome to be able to sort of like spelunk and be able to go to different teams and sort of see that broader picture. and I just don't see a way that you would be able to get that type of visibility that's super interesting at a lower level. I think the other thing is whether it's warranted or not, you do get some amount of status when you go to a meeting. People just listen to you. They listen to your harebrained ideas and it's kind of nice because you don't necessarily have to prove yourself over and over again. There's a bit less professional, not fights, but just establishing that you know what you're talking about.

53:09Yeah, yeah. Now, the bad things are, you know, there's a lot of folks that are really good in tech and being really effective as a principal engineer, but then they also, you know, myself included, they're like, okay, cool. Well, that sort of makes me an expert in pretty much everything. And so you would get these principal engineers together. We had a weekly meeting, and so it would be like, okay, if you wanted to talk about, like, establishing a constitution for a small island nation, all of a sudden they would just be like, well, like here are the main considerations. It's like nobody has a background in government policy.

53:43But all of a sudden, like just because you're sort of trained to do so, you start to like pitch in. You're like, well, actually, you know, maybe we should have two branches of government or three branches of government. And it just sounds like we would know what we're doing, but we don't. And so there's this trap. And again, I've fallen into it many times where you actually think you're an expert in one thing, but you're actually not, right? And so, you know, take LLMs. There's a ton of folks that understand AI. I left before it was sort of like allowed to use internally, but I think you can use it now.

54:17I'm not an expert in LLMs at all, but I do think that the expectation would be that you understand, you know, how they work. But then the expectation's also like, hey, what should our policy be? How should we be thinking about this stuff? And I think that's fine for mature technologies, potentially, like you can ramp yourself up for it. But as that particular landscape is changing so quickly, I think there's this sort of trap where you speak as an authority, even though you haven't had the requisite time to ramp up on something. And you've been there for 17 years at Amazon. What are your favorite parts of the culture?

54:56You know, there's a lot of things that there's values that we all know, like the frugality, customer obsession. What were the things that you found to be the most interesting or the ones that had lasting impact? And how did they change? How did Amazon change over 17 years? They must have changed. No, I think the things I missed the most and the secret sauce, yeah, the leadership principles are good, but I think the actual secret sauce there is principled thinking. Right. And so, yeah. So, you know, there's, you know, invent and simplify and bias for action and all of this stuff. But like ultimately, the thing that is amazing about those leadership principles aren't the specific stances that they took.

55:41So they decided that customer obsession is a big deal. They decided that bias for action is a big deal, all of these things. But really, if you look at a meta level, you'd be like, oh, these guys have principles that they won't budge on. I sort of think about it in terms of math and axioms. Like you just take certain things to be true. You know, two lines that are parallel, if you extend them out to infinity, won't touch them and won't touch with each other. Yeah, you assume that's true. Yeah, you don't prove that. It's an axiom. And then based off of that, you're able to build a system of mathematics, right?

56:14And so it's the same thing with the corporate leadership principles at Amazon. They basically said, okay, we are going to fix these things to be true. There are 16 or 12 or I don't know. They just sort of bolted some on. They were 14 and now they're 16. But there are like four or five that are just really core to Amazon. And we just fixed those things to be true. Which ones were the ones that you felt were the most present? Customer obsession. We are absolutely customer obsessed. We'll just burn money to delight a customer. You can be in a meeting with a VP as an intern and you say, hey, that's a bad customer experience.

56:52It would be like a needle coming off a record. It would just be like, what? What are you talking about? Like immediately, right? You know, bias for action. So like just get some stuff done. Stop asking for permission. Just like go and do it. Right. Ownership. It's just like you own your software. You run the, you know, you do the operations, you know, you own the bug count, all of this stuff, right? So those are the ones that are, like, those are fixed. And then you start layering things on top of it. And I think it's really great. But, you know, you could take Amazon and you could have, like, the, you know, evil goatee version of Amazon, which is just sort of the opposite of those things.

57:26And that would still be a really valid and awesome company. So you could say, okay, well, it's the opposite of customer obsession. It's not customer obsession or not being customer obsessed. I think it's, you know, like being about your staff. Yeah. Which is Google. It could be like, hey, we really care about our people above everything else. Or it could be, you know, let's not mince around it. We care about top line or bottom line revenue. Yeah. That's totally valid. Right. And then you could just fix that. You wouldn't you can't prove that, you know, being, you know, staff focused is a bad thing.

57:59You just build that. And then, you know, a certain set of things will happen. Like great things are going to happen. and then like not so great things are going to happen. Those not great things that happen, you can try to mitigate them, but you can't fix them because you have started with this principled approach to everything. Yeah, yeah. It all goes like everything has. Yeah. I see what you mean, but I think what you're saying is like, it might be less about what the specific principles are. I mean, Amazon has theirs and we know about them, but it's just sticking to them and not keeping wiggling.

58:29Because if you keep wiggling, it's like, what was the point, right? Then you're going to have a really kind of mediocre, truly not standout company, whatever you do. What does it actually mean to be principled and to not bend? It could be really easy to do so. So that's an amazing secret sauce of Amazon's. People look at the leadership principle and I'm like, no, it's principled thinking. And a lot of this, honestly, from what I understand, talking to you earlier and some other people, a lot of it probably comes from Jeff Bezos being from the top down, being very principled, on not not giving not saying we we will do this whatever it takes sounds like it was customer session initially and then some other things yeah yeah absolutely and he's he was he was an absolute genius uh when it came through so i'm a i'm a you know i'm a jeff bezos fan boy um for sure like it just it just worked um another thing that uh that's amazon secret sauce is just the writing culture And so I spent on the order of one to four hours every day reading while I was a principal engineer.

59:29And we had a standard format. It was a six-page memo. And that would be our business strategy. That would be a system design. That would be what we call the PRFAQ, so a press release and frequently asked questions for a new line of business or a new initiative. and everybody was sort of constrained to the six page format. And everybody just produces documents in that format for whatever they need to do. And so when I would try to get up to speed on a particular thing, I would just be like, give me your six pagers, give me all your documents. And I just got really, really good at just reading these documents to get up to speed, which was a self-fulfilling and virtuous cycle, which is just like, okay, well now I need to express myself.

1:00:16And so I will write a six pager and that will set the context for whatever we're working on. We'd go to a meeting, you would read the six pager. And it was just super great to just actually just have people do study hall at the beginning part of a meeting where you just, everybody just gets fast forwarded. And then you have a really great discussion at the end. That is what an amazing culture that I think that almost every other company should replicate if they could. But I think the difficulty would be like you actually have to be disciplined and actually have a reading culture and principled and have a reading culture and then actually value writing.

1:00:55Yeah. I almost wonder if unless it comes from the top, some of these things might just be really, really hard to do. Yeah. One thing that I figured is we're in your studio right now and you have a lot of these blocks and I asked them what they are. Are they for promotions or projects or whatever? They're for patents. Yeah. And this is for patent number 10 ,000, 10 ,824 ,964. Can you tell me about why you have these, how they come about? Yeah. What you needed to do for them? So the highest order bit is like, you know, for better or for worse, there are software patents that exist. Amazon, they'll say that basically the reason they have them is defensively.

1:01:37Because, you know, other people will assert that, hey, you're in violation of our patents or our ID. And then, you know, we'll use them reactively. Okay, fine. But, you know, you're also in violation of these other things. And so, you know, there is a culture of trying to make sure that, you know, we protect ourselves in that way. But, you know, there's the other part of software patents, which is basically like, hey, can you really patent like math or whatever? And so what I learned over time is that, you know, I'm just a really bad IP lawyer, even though, you know, as a principal engineer, I might cosplay as somebody that really understands software patents.

1:02:11Right. And at the end of the day, you know, what we would do is we would take our important six pagers and we would hand them over to the legal team. And then they would just be like, oh, this stuff is really interesting. Like, let's explore that. And so it turned into this awesome thing where, like, we just had ready inputs to go into, like, the, you know, into that particular system. A writing culture turns out has a bunch of benefits. Exactly. Exactly. And I think that there's this sort of like it's the concept is called like the curse of knowledge, which is essentially like if you understand something, you discount how long, like how easy that concept is.

1:02:48And so it's just like you don't get it, you don't get it, you don't get it. And then you get it and then you're like, oh, that's trivial. Right. Even though, you know, there could have been, you know, it could actually be novel or it could actually be interesting. And so what ends up happening is that you would just throw these documents over to the lawyers and then they would basically be like, oh, this stuff is great. And you would just be like, well, that's just regular software development or that's just the context and domain that we were living in. You know, it turns out that there's some interesting stuff.

1:03:15This particular patent I'm proud of. So there's a system design interview question that seems to be popular right now, which is like design ticket master. Right. And so I worked on Amazon tickets and we ended up shuttering that business. But we ended up building one of the world's fastest ticket selling systems in the world. We can do many, many orders per second. So the use case is basically at T0, that's for a really big ticket on sale. That's when the maximum amount of demand and requests are coming in. And you want to sell out all of your ticket supply as quickly as possible. The problem is, I think, one where you have seated concerts.

1:03:58And so when you purchase a ticket, most of the time with the system design stuff, it'll be like general admission or it won't be a high ticket, like one with a bunch of demand. You have to find contiguous seats. Yeah, so the ones that are next to each other. Yes, exactly. And so it's actually really hard. like suppose it was a SQL database as your backing store. Like how do you come up with a SQL query that's just like, hey, give me the best four tickets, you know, within this particular price range that are sitting next to each other? Yeah, now you're thinking, so this is a real world thing where you want to be as efficient as possible in terms of research usage.

1:04:41Maybe you want to minimize your CPU or memory, depending on what you have, I assume. And you need to do this quickly or as rapidly as possible to give this to people. Okay, so now we're talking about a problem that seems like pretty novel in some ways, right? Yeah, and so I did this patent with a senior principal. I was a senior engineer at the time. But the idea is like, what is the theoretical maximum speed by which we could show this inventory to people? And it turns out that even if you have a high ticket on sale, you only have like thousands of tickets at the end of the day. So instead of making a request to like a backend that would conduct some sort of search across the space, what if you actually inverted it and then you basically had each of the individual hosts have like some view on the entire arena or a venue that was there and you loaded up all of that availability and inventory into like L2 cache on a CPU?

1:05:44Yeah. Because it's actually not that many. So if you have this compact representation. Yeah, we'll catch it pretty big, yeah. Then what you can do is you can do bit manipulation to like really, really quickly get contiguous seats that are there. And then what you do is you can like send in that particular request and try to like reserve those particular seats. Now there's a logging problem. Which is much more tractable than like, hey, there's, you know, two million people that have just hit your on this page. And each of them, I'm going to search for each of them. Yes. So the inversion of that ordering process by which you actually send out the inventory to the individual nodes and then load it up into CPU cache and then just do bit manipulation and then try to lock that resource from the individual nodes.

1:06:33That was the basis of this particular patent. Awesome. That's clever. And that sounds like some people are always asking, like, oh, you know, on my job, I don't use the algorithm stuff or any of the formal methods. It sounds like there are some uses of it, especially when you're trying to figure out what is it like when you're just taking away from the pattern, just having a problem like this and saying, like, what is the theoretical limit that we can do? What is the fastest possible? Like to answer that, you probably want to have access to these tools. Like, you know, like, so it's not always a time and effort to actually get into these things.

1:07:08And so what are you up to now that you've you've left Amazon a year ago after like 17, 18 very long years? You know, I'm just, you know, I'm just making content. I'm just sort of living the dream there. You know, making YouTube videos. It started up a newsletter. I've had Discord community and yeah. Yeah. And we're going to link all of those below. I actually got to first know you before we started talking. This was probably a few years ago from your YouTube videos, which you shared a lot about Amazon things, software engineering things, and just your general thinking. But yeah, your newsletter is a new one.

1:07:45So we'll link it in the show notes below. It's always a good way to keep in touch and also on your YouTube channel. Awesome. So as closing, I have some rapid questions. So I'll just ask and you just shoot what comes to mind. What is career advice that greatly helped you in your path? Yeah, I mean, this is, you know, I talk a lot about this. It's kind of like, oh, what's your favorite food or your favorite movie? It's just like there's so much there and it's hard to pick one. What I would say is instead of saying like, hey, what's the technology that I should learn that's really going to, you know, make my career, you know, solid?

1:08:21Instead, sort of flip it around and say like, how can I quickly learn skills? that makes you that makes you sort of like recession proof right that that sort of makes you valuable it's essentially meta learning it's like how can i learn something faster and faster if that's your focus then you'll always be you'll never have a problem finding a job and you'll never have a problem progressing in your career now some of the skills may be difficult to find resources on online but you know i think if you just sort of think about like what's a valuable skill that if I knew right now would, you know, make my, you know, job search easier or would like make me, you know, perform better on the job.

1:09:03And then just sort of thinking about acquiring that skill as quickly as possible. And do it now, like don't wait. Yeah. Well, people tend to postpone themselves. They'll be like, oh, well, I'll start when, you know, everything is lined up. But like to begin, you just need to begin. Like when you start something that only then will you know what you need to do instead of saying like, oh, I need to get everything that I need to do first before I start. You've used a lot of programming languages. Which one's your favorite and why? And which one you do dislike most? Yeah, you know, I have like a, you know, obviously there's no perfect programming language.

1:09:39What I would say is like, I really enjoyed Perl and nobody would ever give that answer. But I just like this concept of like, There's just so many different ways to do it. It's a write-only language. You can't read anybody else's Perl. And it's actually one of the languages that uses up the most power. It's the least efficient. It's interpreted. It's just terrible. Most of Booking.com still runs it. There's some of it. Amazon's back end was, for a long time, it still might be. Sort of like Perlmason is sort of like web technology bolted onto Perl. But I just kind of like it. I just feel like I can express myself.

1:10:17and there's just like, there's just, however you'd like to express yourself, you can. It also looked like an ASCII factory blew up sometimes. And so it's just like, it's, you know, now that it's on a podcast, I wouldn't really advertise that fact. The best programming languages right now, I think Rust is pretty interesting. So I might, you know, pick that up. At the end of the day, like, I really love the boring languages. So, you know, Java with, you know, for all of its stuff, like it's verbosity and I think it's just a great language like a JVM based language um that has essentially like great like library support and a bunch of stuff written for it but it's just like super boring maybe it's just because I'm from Amazon and we do this like enterprise stuff like it's a fine language and then I see you have a large bookshelf here you also read a lot especially at Amazon all the most internal documents what is a book that you would recommend something around software engineering that you enjoyed.

1:11:17And it cannot be that book. It can't be your book. What I would say is, you know, I just given the advice about, you know, meta learning and career growth. I think that most software developers should read a book by Cal Newport. It's called So Good They Can't Ignore You. And so the concept there is around career capital. So like, what are the skills that are in the most demand? And if you can just like learn those skills, then you become in demand. And then, you know, from there you can choose what type of lifestyle that you'd like. You know, you can also like sort of lean into, you know, some of the science of meta learning.

1:11:53So deliberate practice, space repetition, that sort of thing. In terms of like tech books, I think the new AI engineering book by Chip Wynn is amazing. I think DDIA. So the design of data intensive. So good. A new version is coming the end of the year, actually. And I'm excited about that. I think that would be pretty good. But at the end of the day, you don't want one book on your bookshelf. You want 50 books on your bookshelf. And so I think within a particular sub-genre of tech books, I'd have recommendations there. Steve, this was great. Awesome. Really enjoyed it. Yeah, great. Thanks so much for having me.

1:12:34Thanks a lot for Steve for sharing all these details. Although Amazon's principal engineering level feels surprisingly difficult to get promoted to, I have yet to hear of such a strong principal engineering community than what Amazon builds and keeps investing in. This community itself could be a reason enough to consider the company after the principal plus level, should you have the opportunity to do so. For a deep dive into Amazon's engineering culture, including the details on compensation, career ladders, performance reviews and engineering processes, check out the Pragmatic Engineer Deep Dive linked in the show notes below.

1:13:05If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. This helps more people discover the podcast and a special thank you if you leave a rating. Thanks and see you in the next one.