AWS Distinguished Eng: Learning From 3000 Incidents And How Engineering Is Changing | Marc Brooker

23:58Peterman:I mean, since you have worked on AWS for almost two decades. I'm sure you have a lot of experience building distributed systems. And I think one of the most common advice that you hear, I guess this is maybe in the context of system design, is I almost hear almost 100 % of the time people will say, just throw a cache on it. Or you'll have a system design, you say, how do you make it better? Let's put a cache here. Let's put a cache there. And I saw you had a tweet that said that there are cases where caches are bad despite people saying it's best practice. I was curious if you could explain that.

24:36Marc Brooker:Yeah, so caching's good, right? Like it's, hey, I'm going to take these core ideas from computer science of temporal and spatial locality and I'm going to exploit those to make my system faster, scale better, et cetera. And so, you know, obviously very attractive. But the downside of caches, especially in distributed systems, is they have this mode, right? Like they have this, you know, there's a mode where the cache is full and the cache is full of the right data in time and space to perform very well. And there's a mode where the cache is empty or contains the wrong data. And in the first mode, the system is fast and happy and healthy.

25:19Marc Brooker:In the second mode, the system is slow, often down because now the backend isn't scaled to deal with. all of this uncashed traffic, customers are very disappointed. And often it is down in a stable way. And this is this kind of idea of metastable failures where the system has switched from state one to state two. And in state two, it's still stable, right? Like it's still, it's down, but it's not going to come back up under its own energy because, for example, all of this traffic is causing a huge amount of contention in my database. or is saturating the network. And so I can't even refill the cache.

26:00Marc Brooker:It's not even getting the right kind of data in. And so, you know, when I talk about the downsides of caches, it's really about, you know, how do we avoid that modality between, you know, fast and, you know, that value of caches and the, you know, how do we avoid the state where we're down? um and so if i go back to to dsql like our answer there is dsql what we call the storage tier is essentially a cache but it is a complete cache it contains every row in the database um and so it doesn't have this mode where how do i recover from it being empty or containing the wrong data it contains all of the data um similarly if you look at a a more let's say classical relational database design like aurora the aurora leader is constantly telling the potential failover targets here's something you should cache here's something you should cache here's something you should cache so when a failover happens the cache is warm on you know on on the failover target um and so those are the kinds of things that you can do to avoid those modalities but in general um you know and i wouldn't extract this as a rule or or or say that you know this applies 100 of the time but in general i prefer to see the teams around me avoiding caching where possible i prefer patterns where you have a it's a complete materialized view of the data if you need very fast access to it especially if it's slow moving just pull it down onto your local machine and work with it in memory you know if it's only being updated once a week who cares like just make lots of copies of it um uh so that's that's one pattern or you know use a scalable back end you know dsql or dynamo db or whatever your favorite scalable database is and keep your database vendor honest about getting to the the scale and performance you need rather than putting a cache in front of things So caching isn't a bad pattern, but it is a pattern with some significant downsides that are really best avoided.

28:16Peterman:In practice, how often do you see that metastable failure, though?

28:22Marc Brooker:Yeah, you know, it's not super common, right? Like you might go years without seeing something like that. But if you look across the biggest, most impactful system postmortems across the industry, I would say that these kinds of metastable failures have been an underlying cause in probably a majority of them. and it's super important that you know as an industry and as a community of practice we understand those things deeply because the also those cases where these do happen you know tend to be larger scale issues longer recovery time issues and and and more complex to fix issues right where you have to often you know turn it off and turn it back on again which is this very very painful thing for a team or an organization to do.

29:18Marc Brooker:And, you know, and so again, like you might go here, it's operating a system with seeing nothing like this. But if you look at the most impactful issues, it's actually fairly common as an underlying cause for those issues. And so, you know, it's kind of both of these things are being quite uncommon and being rather common.

29:36Peterman:I was reading your blog and you have a series of posts on how AI may impact the future of software engineering. And I kind of want to pick your brain on that. So what's your perspective on how you think AI will impact software engineering and how it will change things?

29:54Marc Brooker:Yeah, I mean, it's, you know, hard, maybe harder than ever to tell the future. And so, you know, this is a set of maybe guesses and predictions about the future. So I'll say the first thing I deeply believe about software is we have only just started to see the impact that software is going to have on the world. There is such an opportunity for more software to exist, bigger software, better software, more personal software, all of these things. And so software has, throughout its 60-ish year history, been supply constrained. And I think that's going to remain true. I think the opportunity for software in the world is just almost unbounded.

30:47Marc Brooker:And that's really exciting, right? It's really exciting to be at a moment when the economics of building software are changing and are changing rather quickly. and that gives us an opportunity to think about what could we do in the world with a lot more software you know a lot more software personalization a lot more just the right software in the right place at the right time and you know that gives me a huge amount of you know excitement about the future of this industry because we have a massive opportunity ahead of us, driven by these changing economics of software development. Now, also with those changes, there are going to be needs for us as software practitioners, people who build software, people who love software to adapt.

31:46Marc Brooker:And that means that software careers are going to look different. They're going to look different early on. They're going to look different later on. I think the software business is going to look different. And the success of people and organizations over the next, you know, next, who knows, five years, decade, is going to be largely predicated on their ability to adapt to that change and to lead that change.

32:18Peterman:You told this story about this guy who bet on analog circuits when obviously we know digital became kind of the more dominant way. Yet he made good money. For the people who maybe don't want to adapt, you could still get by and succeed. It's not going to be like a crazy thing. Is that kind of the takeaway and why you brought up that story?

32:41Marc Brooker:Yeah, I think that's the right takeaway. And so if I sort of break down the the world into three tiers, I think there's going to remain a huge amount of joy in the craft of software, like the craft of joinery with hand saws. It's a nice way to spend time. It's not a particularly economically interesting activity anymore, but not everything we do has to be an economically interesting opportunity. It can just be something I do because I enjoy it, because I I enjoy the product of it because I enjoy talking to people about it. Right. And so there's, you know, I don't think that is going to go away.

33:22Marc Brooker:I think we're going to see, you know, a lot of interest in that. Like there's been interest in retro computing and, you know, people who run an Apple II as their desktop. And like, well, again, it's wildly impractical. It's not economically interesting, but it's fun and something I, you know, could do as a hobby. And so, you know, that's going to be a remaining part of the world of software for probably forever. And then there's this, you know, kind of story that I told in the blog post. And I think this relates to, you know, driving change in the real world, it's always harder than it looks from the outside, right?

34:01Marc Brooker:Like, as you get into the details, things become more difficult, they become more dependent on people, they become more dependent on politics and policy and, you know our various irrationalities as humans and and so driven by that you know there is going to be a huge amount of and a shrinking over time amount but but a huge amount of the software industry that is run in what i might call the old way right past techniques past languages past technologies. And there's real economic opportunity in engaging with that part of the world. As we saw with analog electronics, analog electronics still very much exist.

34:48Marc Brooker:In fact, there are parts of the world like radio and power systems where there's been incredible technological advancement in those fields. But they have become more niche. And so digital became the mainstream. We wouldn't be talking like we are today if it wasn't for this, you know, 12 orders of magnitude or whatever explosion in digital transistor counts.

35:16Marc Brooker:But there's interesting opportunity there. And I think that interesting opportunity is going to change shape and become more and more specialized and more and more niche and great careers to be built there. And then there is the mainstream, which I think is going to adopt these new technologies from agentic development to AI-powered development to specification-driven development and a whole lot of other new things whose names we don't even know yet to build software at a speed and a cost that is unimaginable to do with the old techniques. and I think that is where correctly the majority of the industry is going to be going.

36:03Marc Brooker:I think that's where the majority of careers are going to be built. I think that's where the majority of economic opportunity is. It's the space I'd be in if I was building a company today. It's the space I'm in in my role and the one I would sort of personally be most excited about. But yeah, it isn't the only one. I think there's going to be the spectrum of software practice and And especially where software engages with the physical world, there are going to be some really interesting questions about how do we bring these new technologies, how do we bring these new practices into the various many niches that software is going to and has over six decades kind of wormed its way into.

36:49Peterman:It's interesting. You mentioned joinery. I wonder if down the road we will see apps on the app store that people pay extra for because it's marketed as this was written by a human or it was written by hand. It's a bespoke custom app. Crazy how the world is going to change. But so it sounds like change is obviously the common case. It's the one that we should be thinking about. Maybe we can break up the conversation in two parts. One is for junior engineers, what is important given that code is kind of flowing like water now?

37:26Marc Brooker:At risk of being a bit meta about our past conversation, it really is about finding those problems that matter and doing that early in a career. And, you know, that requires an understanding of customers. It requires an understanding of the business. It requires an understanding of economics and of systems. and that can, I think that's going to move from being almost kind of senior engineer work of like, oh, well, now you're going to go and talk to customers and actually understand the context of the stuff you're building to being more and more part of even the earliest steps of an engineering career, right?

38:09Marc Brooker:Like here's the context, here's the problem, here's the customer, let's go off and work together and solve this problem with all of this context. And I think that's going to be super exciting for one set of folks and a little bit frustrating for people who have come into looking for a pure software development career, right? Looking for a career where they sit down, open their IDE, start typing and and don't stop for eight hours i think that's going to be a mode that we're going to see fewer people in and a mode that's going to be harder and harder to build a career around now the other mode of oh i'm excited to go off and learn from my customers about what they're building and what they need i think that's going to be ever more highly you know highly valuable and so super exciting opportunity to build you know build careers there and then maybe and this might come across as being a little bit, you know, paradoxical, I think there's also a ton of opportunity for, you know, folks who are extremely technically deep, you know, who are, you know, deep on optimization problems, or deep on infrastructure problems, or deep on, you know, various scientific things, or deep on databases, or deep on, you know, one of the many, many topics that are behind our industry because I think the ability to ask the right questions is also much more valuable than it has ever been.

39:49Marc Brooker:And so I think there is a ton of opportunity for people coming into the industry with deep technical or scientific knowledge to now leverage that in ways that maybe were hard before. There was too much sort of boilerplate to really use that leverage that you have. And so I think we're going to see a lot more of those kinds of careers of really kind of building expertise in a technical topic, in a scientific topic, and then be able to turn that into software and software products in a way that was really difficult before and in some cases wasn't possible before and is now vastly easier.

40:31Peterman:If I was to look at a career ladder's expectations, some of what you described of maybe engaging with the customers and understanding the business context. Uniquely in software engineering, it feels like the earliest levels are insulated from all of that. You have your tech lead, tech leads handing out tasks, and then the early level engineers just given tasks, just converted into code. And it sounds like that part's relatively solved and if not now maybe i'd be surprised if a year or two from now wasn't like completely solved um and i i think that could scare a lot of junior engineers because they would think you're going to expect me to graduate from college or start working as software engineer and then i would have the senior engineer expectations what would you say to the the scared software engineer that's just entering the industry with all this change?

41:30Marc Brooker:Yeah, you know, I think, well, I would remind them that, you know, we as people who hire and build organizations of software engineers, and they as people who have are building software engineering careers have really aligned incentives, right? Like, you know, it's not valuable to hire a bunch of people and set them up to fail like that's nobody wants that it's it's it's not an outcome uh that is good for anybody and so yeah we're going to need to figure out how do you support people on on that path how do you help people learn those things how do you give them the right guardrails you know hey that first time that you go out and talk to a customer yeah it's going to be scary.

42:16Marc Brooker:My first time talking to an AWS customer, it was super scary. But I got a bunch of help with that. And I got a bunch of advice. And I got a bunch of mentorship. And I got a bunch of feedback. And I got better and better at that over time. And I think that's exactly what these things look like. You start off and you start small. And you learn you know as you go and and and so that feedback loop goes faster and so i don't expect that people coming in from college or you know will will come in with all of this knowledge i think you know it's never been true that people coming into technical or engineering careers straight out of college know everything or any career for that matter right like you talk to to teachers about you know what they've learned on their job versus what they learned you know studying uh you know they learn a huge amount in things like internships and so on and over the course of a career or doctors or anybody in, you know, in a field like that.

43:19Marc Brooker:And so, yeah, it is going to be about learning. And I think the emphasis on what people learn is going to be different. I think it is going to require, you know, leaders like me who, you know, care deeply about, you know, hiring and developing folks early in their career to be really thoughtful about what, you know, what does that new ladder look like? And, you know, we're doing a lot of that thinking. I think people are doing that kind of thinking across the industry. And yeah, it's changing fast. It's uncertain. It's an interesting time to be graduating. But again, like it's a super exciting time.

43:58Marc Brooker:I think

43:58Peterman:that's just the scale of the opportunity is bigger than it's ever been. It sounds like your advice for senior engineers is different from that of junior engineers. What is your thinking there?

44:09Marc Brooker:Yeah, I mean, I think for folks there, the challenge is how do you retain the value of this incredible experience and knowledge that you've gained over a career while not falling behind, while learning how to best use the tools? And when I look at senior folks, this is a challenge ahead of them. I think a lot of people have found themselves in influence and leadership type positions where they aren't hands-on building every day. And I think it's going to be harder and harder to be in that kind of role and be able to influence and advise in a relevant way, in a positive way. um and so really i think my advice for folks is is is you kind of got to get building like you've got to get back in get back into it you need to deeply understand how the practice of building software and the practice of designing software has changed and and is continuing to change and and so the challenge is how do i you know really take advantage of all of this knowledge and expertise that I've built up in my career and be super curious and be super hands-on and really be in the details.

45:36Marc Brooker:And the good news for that, well, I think there's two bits of good news. One of them is because of these new tools, time spent as a practitioner is so much more leverage than it is today. You can build such cool stuff during that period of time. the amount of kind of wasted time and boilerplate and so on is so much smaller. And so you really do have this opportunity. And the other one is, again, like, why did, you know, why did we get into the space? Well, I didn't get into it so I could go to meetings and sound smart. I got into it because I love learning and because I love building technology and because I love, you know, solving my customers' problems and because I love, you know, learning about new, you know, new technologies and learning new things.

46:24Marc Brooker:And there's more opportunity to do that than ever before. Again, because of this new set of tools and the leverage that comes with them. And so it really is getting back to, why are you here? Why did you get into this career? And I think it really gets us as technology-focused people closer to our original answer to that. It's really obvious to me right now when I speak to practitioners you know who and who isn't using a you know modern set of agentic powered developer practices right and the people who are um have these really interesting things to say about the strengths and weaknesses of those approaches and the work that still needs to be done and the integrations that still need to be done and the things that are working and aren't uh and the people who are you know, using them hands-on, have such a poor mental model of how they work, what they're good at, what they're not good at, that the things they say about them tend to be essentially fiction.

47:31Marc Brooker:And so, you know, I think we are in this minute that if you aren't doing it hands-on, your opinion about it is very likely to be completely wrong. and that takes a level of humility to admit that you know is is tough you know it's tough for folks with fancy titles and it's tough for folks with with distinguished careers uh but i i think it's

47:54Peterman:a must i i feel like there's a common sentiment among software engineers when they when they work with someone who is a quote-unquote tech lead but they're not really hands-on so they've kind of been in the docs for the last five years or or so and there's these minor things they can tell that this person doesn't actually understand the underlying thing and sounds like that gap will widen with these new tools which is if you're you're looking at things from a thousand feet up and you're not actually using the tools that's just another thing that separates you from the people who are actually building where you'll be very out of touch and i i think uh you know when

48:37Marc Brooker:I look at, I think that's always been true. I think it is wider than, you know, ever before. And, but when I look at the, you know, engineering leaders that I've really respected and learned a huge amount from over my career, you know, for example, some of the folks who built S3, you know, 20 years ago, that was such a successful product because those folks were so deep in the details and so grounded on the use cases and so deep in the economics and really just did you know really thought about both the kind of strategic world of like how is this cloud thing going to change the way people want to interact with storage but also the you know minute-to-minute details of what's fast now what's slow what's good what's bad and I think when you think about an extremely enduring product like S3 or EC2, I think it's been that groundedness in the details from early on, from all levels of leadership that has made those things so successful, where other products seemingly with the same amount of early promise didn't turn out to be as successful.

49:59Yeah.

49:59Peterman:I think one of the last topics that I wanted to ask you about was writing. You have a ton of awesome posts on your blog. The style of writing is incredibly clear. And I was curious, why do you write so much as an engineer?

50:19Marc Brooker:Writing and speaking, but especially writing, have this incredible power. and you know for for technical folks it's this incredible multiplier and being able to take these ideas that's in your head and share them with the world and you know you can can take a set of technical ideas in your head and share them with the world by building a great product and that's a fantastic thing to do you can share them in the world kind of one-on-one you know mentorship teach people learn small groups also a great way to spend time but the multiplication factor of doing a talk or even more of of writing something is so much higher right like there are so many more people that you can share that with and it lasts for a much longer period of time and so just having something written on my blog even that I wrote like a decade ago that I can share with someone and say, here's how to think about this problem.

51:22Marc Brooker:Here's an insight that I wanted to share with you, or have people discover that organically is just super powerful. And so writing lets you scale out the impact of your expertise in space and time in a way that's really hard to do in other media. I think with video and with podcasts and so on, we've seen other ways to do that but i think writing remains kind of uniquely powerful and then there's also this idea which is this kind of core belief culturally at amazon and i've obviously been affected by this over the years that you know writing forces a level of mental clarity that speaking making slide decks etc doesn't and you know that's something that has also really been my experience of sitting down to write something down forces me to think that through at a depth that I wouldn't have been been forced to think it through without that um and so I saw one of you know your early conversations with was with Leslie Lamport who kind of takes that a step further and say hey you know it's formal mathematics that is the next step there and I I love that point um but I I think writing is this really accessible thing for for people to do that does force a level of thinking and so i do a lot of writing sometimes just for myself right like i'll write a doc not ever intending to share it with anybody but just to sharpen my own thinking on a on a particular point and so some of that combination of three things right like i just i just have something to say and i want to say it um you know I have something to say and I want to scale it out in time and space.

53:05Marc Brooker:And I want to sharpen my own thinking on a subject or the thinking of a small group on a subject in a way that writing is just a super powerful tool to do.

53:20Peterman:Definitely. Yeah, I remember being surprised early in my career. I had a manager or tech lead who we would write these docs on either designs or the strategy. And he said, even if you just wrote it and you threw it away, it would still be worthwhile because you'll realize things as you're writing and that clarity will save you a lot of time down the road. And it's interesting to me because a lot of engineers, they complain about writing docs. Docs and all the stuff around the code, they kind of hate that. It's a sign of slow, big company processes and what would you say to an engineer like that who's saying, just let me write the code?

54:03Marc Brooker:Yeah, and that's a great point. And I think it really depends on the level of problem you're trying to solve. And so if I look at, I'm going to pick on UML for a minute here, right? Like it's a sort of semi-formal software design process and not one that I've ever found useful because I think it just happens at the wrong semantic level. I think it's bothered with details at a level that aren't helpful. And I think a lot of the let's go off and document this has a similar problem, right? Like, does this actually require that level of reflection and thinking? And so I think what, you know, for me, separates a valuable doc writing and thinking process from a busy work process is understanding what you're getting out of it.

54:55Marc Brooker:and what you're getting out of it might be an artifact to share with the future, which is super valuable. Either your future self, if you've got a terrible memory like me, or new teams, new people, or I want to share something with customers or I want to share something with the world. And so that's super valuable. Or I want to write down something so I can think through a really difficult, often one-way door kind of hard-to-change technical decision or API design decision. And I'm not going to do that every time I make a technical decision. It's not worth it because a lot of those technical decisions are either easy or not as critical or can just be taken back if we figure out they're wrong.

55:43Marc Brooker:But I am going to spend my time that way when there are key decisions to make when there are key um insights to to find and i think you know and so it is that like what is the purpose of writing uh that uh that that separates well-spent time from poorly spent time now there are people who still don't like writing even when it's well-spent time even when it's like you know you have to explain this piece of technology to you know to a future team um i think that's a skill worth developing you know sometimes you you do need to you know uh eat your vegetables uh you know and it's it's it is a skill worth getting good at um and you know especially in documenting the core kind of technical decisions behind a design is is so useful.

56:45Marc Brooker:And that's useful in two ways, by the way. One of them is, as we think about building a big system, we make thousands of decisions. And some of those decisions are very carefully chosen, very particular, and very impactful. And some of those decisions are the best thing we could guess in the moment based on having no data to make that decision. and it's super useful for people who are coming in to improve that system down the line to be able to look at the design and say which of these things were very carefully chosen and thought through and which of these things were arbitrary and because the arbitrary things like okay well I'm going to change that and I'm going to just go ahead and change that because I have better data now I've watched the system run I can go and change those and these other ones are like well well, let me really engage with the reason that we made this decision.

57:39Marc Brooker:Maybe it was non-obvious. Maybe there was some more advanced thinking. And so being able to kind of understand the amount of thought that went into a decision is almost as important as understanding what that thought was.

57:50Peterman:You had a really interesting blog post. This was from a while back. It's titled The Four Hobbies and Apparent Expertise. And you introduced this really interesting idea. It's a two-by-two matrix. And on one side, there's doing versus discussing. And on the other side, there's the hobby and the gear.

58:13Marc Brooker:Maybe I can overlay it for people who want to see.

58:16Peterman:And then later, you kind of liken that to your career and how, I guess maybe we can imagine the hobby is actually coding. And maybe the gear is, let's just say it's like your dev setup or something like that. You talked about these two aspects of being in, depending on which quadrant you are, which is there's this tradeoff between expertise and visibility, where imagine you're really into coding and you're really into doing. You're going to be phenomenal in terms of expertise, but maybe not as visible because you're not talking with everyone about how cool your setup is and all of that. But on the flip side, if you're really into the gear, or maybe you're set up in this case, and you're really into discussing, you're on all the messaging posts and that, you might not actually be that good at coding, but you're very visible and you have this apparent competence.

59:11Peterman:And I thought that trade-off was really interesting because I've seen that so much in software engineering too, is there might be someone who's a really quiet coder. They never write anything, but they know everything. because they've just been in the weeds all the time. And then there are people on the complete opposite end of the spectrum that writing all the time, speaking all the time, but maybe not actually practicing as much. And my question to you is, how do you strike that balance? Because obviously too far in either direction is not optimal. So how do you strike that balance?

59:44Marc Brooker:Yeah, that's something that I reflect on a lot. And I do explicitly think that sort of being 100 % on either of those ends is a failure mode. And I think, you know, I will say that I have a lot more personal enjoyment working with the people that are 100 % on the doing side and 0 % on the talking side. I appreciate and deeply, you know, deeply love their expertise. But I do think that, you know, they could have more impact and leverage if they, you know, swung a little bit away from that. You know, I tend to not enjoy as much interacting with the people who are 100 % on the speaking side. But I think they would, you know, have a lot more relevant things to say, you know, if they, you know, swung a little bit back towards the center.

1:00:45Marc Brooker:The other challenge of being on 100 % on the doing side, it sort of gets back to that, how do you find the really important problems? And, you know, if your head's down in your IDE all day, you could very likely be working on the wrong thing. You know, something that isn't as important, isn't as impactful, you know, doesn't have these properties that people want. So, you know, how do you find the optimal balance? I don't have a recipe for, you know, what really is optimal. I tend to do about, let's say, 75, 25 kind of practitioner versus teaching and communicating, maybe 80, 20 at times. I found that's about what feels right for me.

1:01:35Marc Brooker:I would say that there are great people I work with from sort of 90, 10 on that scale up to about 50, 50 on that scale. I think outside of those, folks tend to get into trouble as practitioners. There are people whose job it is to be communicators, and that's great as long as they have the curiosity and are clear about what they know and don't know. But I found that sweet spot at that 75-25 point in my career, and that's what's worked for me. I think in this moment where things are changing so fast, there's so much to learn, swinging a little bit more towards the practitioner side I think generally will help people.

1:02:28Marc Brooker:But again, you don't want to go too far that way because then you lose the what's important for you that comes with interacting with the outside world.

1:02:37Peterman:on the doing versus discussing axis, I kind of view the doing one as if you were too far, you would be underrated. And if you were too far on the discussing, you would be overrated. And for someone who's structuring their career, would you say it's better to be overrated or underrated?

1:03:00Marc Brooker:I think long-term, if you're using that terminology, it's probably better to be underrated. I think being overrated can feel great in the moment, but it's really sustainable and really sort of gets you to where you need to be. I really enjoy things like sports and these sort of creative hobbies and crafts because it does turn that, let's say, perception and reality knob to very much reality, right? Like as a sports person, you can't fool the world for very long. It very quickly becomes very obvious who can and who can't. I think as a craftsperson, the same, right? It very quickly becomes obvious who can and who can't.

1:04:01Marc Brooker:And I think it takes a little bit longer in a field like ours where there is so much kind of qualitative stuff that goes on. But I think long term, when I look at, you know, careers that I really admire and people I really admire, they tend to be people who are personally very honest about their level of knowledge and understanding and skill.

1:04:23Peterman:So people who walk the walk, not necessarily talk the talk.

1:04:27Marc Brooker:I see.

1:04:28Peterman:Yeah, about engineers that you admire, I'd be curious because you have worked at AWS for such a long time, and you have seen so many legendary engineers. Who at AWS do you look up to and why?

1:04:43Marc Brooker:yeah i mean you know just just fantastic one of the blessings of working at a place like aws is i get to work with so many great people um you know maybe because he's retired i'll i'll talk a little bit about so elver murlen uh was was one of the sort of early engineers at aws and um original say huge contributor to the design of s3 a really big contributor to the design of our lot of a lot of our database services over time um al was actually the the cto of amazon for a for a period of time when he realized i think that wasn't the job he wanted to do um but i you know what i really admired about uh about al from early in my career is you know very clearly he was somebody who deeply understood the things he was he was doing and he could work in these two modes I have a great memory of 2010-ish arguing with Al about some of the edge cases in the Paxos paper.

1:05:50Marc Brooker:He was super deep at that level but could also get up to the executive level and talk about cloud strategy and the way we should be explaining things to people and some of the fundamental things that we need to be building. and I really admired that ability to work sort of almost at every level and I was like wow you know this is this is something I aspire to and you know want to model my want to model my own career after and so you know that's I think that is you know the kind of person I've really you know, really enjoy working with is people who do have that, you know, do have that breadth.

1:06:34Marc Brooker:And I think, you know, one of the other things that is really admirable about a lot of these folks is, you know, they don't want to be celebrities, right? They want to do cool work for, you know, have an impact, do great stuff for customers, you know, optimize for having impact.

1:06:51Peterman:For people who want to continue their engineering education and really remain on top of things, deeply understand the technology, do you have any top technical book recommendations?

1:07:03Marc Brooker:You know, anybody who's building distributed system things, I highly recommend Martin Kleppman's book. I think there's a second edition of that coming out soon. There's a new edition of Quantitative Systems Design book, which I also think is great. Right. Hennessy and Patterson's computer architecture book. This is a super, super useful one that covers a ton of ground. I read a ton of, you know, fiction and nonfiction and mostly papers when I'm reading technical things. I find, you know, I find engaging at that level, you know, more useful for me. um and by the way that's that's become way more uh accessible now you know one of the great ways to dive into a paper is you know hey uh you know hey claude uh summarize this for me and then then i can dive into it and and read you know the author's words and i find that mode is great and it's super accessible for people who haven't been able to read papers uh in the past um but uh you know and then there's also a ton of insight in in some really old stuff too uh for example um you know some of the algorithms that we used in in in lambda uh to to manage traffic and manage bursts of traffic come from erlang's work like a hundred years ago on on managing telephone call centers and his book about that.

1:08:36Marc Brooker:And so, you know, folks also shouldn't think that, oh, well, the industry is changing super fast. And so I should only read recent things like this full incredible insights in some of the, you know, older work and in the foundations of computing and infrastructure and networking and computer science that there's, you know more again more maybe more leveraged than ever before you know deeply understanding

1:09:04Peterman:those topics and then last question for you is if you could go back to your younger self when you just joined AWS and give yourself some advice what would you say I think maybe be a little bit

1:09:17Marc Brooker:bolder I really loved the team that I worked with and and you know you know especially in EC2 and in the early days in EBS. And I think I was a little bit more hesitant than was optimal about leaving those teams and looking for the next thing. You know, as, you know, my own learning and impact kind of, you know, tapered off a little bit in those places. And so, you know, I think I've changed organizations kind of, in a big way, four times in my career, and maybe five or six would have been optimal. Not a lot more, but some more. And so don't hesitate to think about what am I learning and who am I learning from?

1:10:03Marc Brooker:And is there a better environment to do that more quickly and to learn more things? And I'm personally highly motivated by being able to follow my curiosity. And every time I've done that in my career, I've found that a valuable move and something that I've personally enjoyed.

1:10:27Peterman:Awesome. Okay, well, thank you so much for your time. I really appreciate it, Mark. Thank you for sharing with the audience. This has been super fun. Thanks so much. Thank you for listening to the podcast. It's a passion project of mine that I really enjoyed building. Another passion project that I've been working on kind of in secret is building an ergonomic keyboard that I wish existed. And I finally have a prototype, so I'd love to show you what we've built. It's ultra low profile and ergonomic and I couldn't find anything like it on the market so that's why we built it. I'll put a link to the keyboard in the description.

1:11:01Peterman:You can take a look and learn more about the project there. We could definitely use your support. Also, if you have any feedback for me about the show, I'd love to hear it. Comments on YouTube have led to guests coming on like Ilya Grigorik and David Fowler. I wasn't aware of them until someone dropped a comment. Also, feedback in the comments helped me learn to reduce the number of cliffhangers in the intros. So your comments definitely make a difference. Please keep letting me know what you'd like to see more of in the show and I'll see you in the next episode.