


Building resilient distributed systems, including why microservices are “architecture of last resort,” how to choose specs vs code as source of truth (“production is truth”), and practical guidance for AI-assisted software delivery. It also covers resilience as four dimensions: robustness, rebound, graceful extensibility, and sustained adaptability, plus “cognitive depth vs cognitive surrender” when using AI.
Sam Newman is the author of widely read microservices books (including Building Microservices) and Building Resilient Distributed Systems. He worked at ThoughtWorks (an XP-influenced consultancy) and was involved in early continuous delivery/build pipeline work; he also spent time teaching automated testing practices to Googlers. He has ~10+ years focused on resilience/distributed systems and has been in the field for decades.
Written by AI. May contain mistakes.
Sam shares his early experiences and interest in technology.
“This episode is presented by TurboBuffer, the AI retrieval layer built on object storage.”
Sam shares his early experiences and interest in technology.
“I've become a bit of a millennial and I think I watch a lot of shorts rather than watching the whole things now.”
Sam discusses his education and valuable industry experience during a sandwich course.
“Living in the UK, we had at places an access to computers quite early on.”
Sam recounts a formative experience with Fortran and the excitement of coding.
“And I got to do work for a company called, back then they were called GC Alstom.”
Sam reflects on his time at ThoughtWorks and the innovative culture there.
Discussion on the role of consultancies like ThoughtWorks in driving innovation.
“And then you eventually made your way to ThoughtWorks where you spent a bunch of time.”
Explore how ThoughtWorks fosters innovation despite being a consultancy.
“But with ThoughtWorks proves to me, yourself and other people, it's actually you can have, like when you are the best in what you do globally, you can actually have a lot of innovation happening there as well, right?”
Discuss the misconception that programming is solely about writing code.
“So much of our working conditions as a consultancy who mostly did co-source delivery at that time, we were doing kind of co-source delivery, a bit of enablement through delivery.”
Understand software development as a collective endeavor beyond coding.
Learn about transitioning from corporate to independent work and its challenges.
“You have written the book, Building Microservices.”
Delve into the origins and rise of microservices in software architecture.
“I mean, honestly, I wanted to write an animal book.”
Clarify the key characteristics that define microservices versus SOA.
“a lot of startups started to say like, you know what, let's just do microservices.”
Examine how Uber implemented microservices to overcome its monolith challenges.
“that's fine do something else then but and there are some opinions so then what is the difference between microservices and what was the other service?”
Learn how Uber transitioned from a monolith to microservices and the challenges faced.
“loan service was on the path of like some really, it needed to be requestable as a really high tier service because a lot of others were using it.”
Understand how microservices facilitate organizational autonomy and their proper use.
“I mean, it's also though, but this is the reality of microservice adoption is like, it's just a tool in a toolbox.”
Explore the differences between tech companies and enterprises in software processes.
“get to have to say whether or not it's a good idea and I think that's where microservices can be especially useful I mean there are other exceptions as well and so an organization like you know, Uber.”
Discuss the debate over the source of truth in software development: code vs. specs.
“I wonder if we will have a different but similar problem of the same thing now that, you know, we know AI tools are here.”
Learn about the role of tacit knowledge in AI-driven software development.
“where they didn't even ever look at the code and they just look at the spec.”
Understand the importance of defining what 'good' looks like in software projects.
“Chain Guard are a proper company, right?”
Understanding what constitutes a 'good' system is essential for developers.
“They're always very narrow and functional requirement stuff.”
Sam Newman discusses his journey and motivations for writing about resilience in distributed systems.
“And with this, let's get back to Sam and what QA and S3 folks already know about running code that they didn't write.”
The intention behind Sam's book is to provide practical guidance for building distributed systems.
“And so I'd go back and try and reread that paper every six months or so.”
Emphasizing the importance of learning from past mistakes to improve system resilience.
“I mean, engineering is all about making sure as many things go right as possible.”
Discussing the role of transparency and honesty in handling incidents and outages.
“And so this, again, is my attempt to sort of highlight all the awesome work that's gone before, the fantastic post-incident reports of people that some companies share.”
Sam outlines three fundamental rules that govern distributed systems and their challenges.
“So I really tried to distill this down because I could never remember the eight, the rule of eight that came out of Sun.”
Understanding why observability is essential for managing distributed systems effectively.
“And in the book, after you introduce these rules, you start, as we mentioned, the second chapter is observability.”
Learn the importance of observability in distributed systems and how to achieve it.
“coming out of that program, therefore give you a high degree of observability.”
Explore how metrics like latencies affect user experience and system performance.
“Charity pays me in stickers, just to be really clear.”
Understand how SLIs and SLOs help in evaluating system resilience.
“okay, can we at least look at a flow of calls?”
Discuss the trade-offs associated with cloud strategies and system resiliency.
“So then the question is, how much do you need?”
Learn how resilient systems adapt and the importance of aligning with user needs.
“You need to know what's happening in all these dimensions that, you know, you care about, like, how long will it take to build?”
Discover the concept of idempotency and its significance in distributed systems.
“But basically, an idempotent operation is one that you can, you know, you can apply that operation multiple times without it actually having an effect multiple times.”
Explore practical examples of idempotency and strategies for implementation.
“A really good example of an idempotent operation would be pressing a lift button.”
Learn about the importance of idempotency in retrying operations safely.
“Rule number two meaning the thing might not be there.”
Explore different strategies for implementing idempotency keys in systems.
“If anything needs to be item potent, do that from the beginning.”
Understand the thundering herd problem and its implications in system design.
“if you use them both together, that can be really useful.”
Learn how business context influences technical decisions and trade-offs.
“And so although they look the same, the mitigations for Eton was a little bit different.”
Learn about handling system partitioning and decision-making in resilience.
“but fundamentally that's what this is about Right, when you have a partition, that's what P in CAP really defines.”
Discover the four key concepts of resilience engineering as proposed by David Woods.
“by coming back to that paper by David Woods I mentioned that was impenetrable to me at the start.”
Explore the importance of robustness and recovery strategies in system design.
“Then there's rebound, which is when something does happen and goes wrong.”
Learn how systems can manage unexpected challenges through graceful extensibility.
“techies are with me, and then we go into the next two, which the third one is graceful extensibility.”
Understand the continuous process of learning from incidents and adapting systems.
“And this learn takes us from graceful sensibility, which is dealing with surprise.”
Discuss how AI impacts software development and the importance of resilience.
“And so the last three chapters of the book, we focus on that, is looking at some of the people and the social aspects of that.”
Examine macroeconomic factors and tactical decisions that affect resilience in AI.
“OpenAI and Anthropik and the hypervisors, right?”
Delve into the limitations of large language models and their lack of causality comprehension.
“What are the things that we know that we do not know about AI?”
Explore the role of AI in coding processes and its limitations.
“And they have trained these models to default to coding.”
Understand cognitive surrender and its effects on critical thinking with AI tools.
“Yeah, cognitive depth is a kind of term that's been around for a while.”
Discuss the implications of AI potentially adding complexity to development processes.
“Yeah, but it is this idea and it's become shorthand for I didn't really look at it.”
Learn about the benefits of modular systems in integrating AI effectively.
“because Kubernetes became popular and it's not developer-friendly.”
Examine the concept of software factories and how to collaborate with AI safely.
“approach is have the architecture in your head, know how your system works, know how the boxes work, know where there's AI inside it.”
Gain insights into resources and strategies for improving system architecture skills.
“There are some companies I know that are already all in on this.”
Learn how to modularize existing codebases for better architecture.
“He's also developed some skills in Claw to take some of those ideas.”
Discover where to find Sam Newman's book and related video content.
“Yeah well I've been doing some stuff on Dave Farley's Modern Software Engineering channel it's kind of a good excuse to do some sort of semi-ranty you know monologues to video and have some fun chats.”
Find out about Sam Newman’s consulting services and insights on resilient systems.
“I do like customized training courses and a whole variety of things around resilient systems and security and microservices.”

Get the gist, explore an idea or ask about something said.
On The Pragmatic Engineer, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Sam Newman is the author who wrote probably the most widely read book on microservices. And yet, as you heard, he said that microservices are an architecture of the last resort. But why? Sam was in the room when the term microservices was coined. I find him to be one of the most thoughtful engineers on this topic, as well as on resilient systems. And his latest book, Building Resilient Distributed Systems, is out now. Today we cover specs versus code as a source of truth and why Sam says that production is truth. item potency keys versus fingerprints, and why you should avoid item potency keys when possible.
0:33The four dimensions of resilience, robustness, rebound, graceful extensibility, and sustained adaptability. Cognitive depth and cognitive surrender, and Sam's advice on how to better use AI to avoid these, and many more. If you want to understand what it takes to build resilient systems, from an expert who has been knee-deep in this topic for 10 plus years, this episode is for you. This episode is presented by TurboBuffer, the AI retrieval layer built on object storage. It's fast, cheap, and extremely scalable. Antropic, Cognition, Notion, and many others are happy customers. Before we start, let me mention Tire, our seasoned sponsor and our product and team I'm very bullish on.
1:08You probably already know that Git hosting has a problem. AI agents. Git is increasingly becoming a bottleneck for modern agent-heavy software development. Engines like you or me are creating more code with agents, those agents push more code, and most of us are running parallel agents, which push way more code. GitHub is clearly struggling to keep up and has frequent outages. So what's the solution? Entire was founded by GitHub's last CEO, Thomas Dumka, who, by the way, was a guest on a podcast when he was still at GitHub. And Thomas revealed Git hosting for the agentic era from scratch. Entire was built to be very fast and to have your repos regionally close to you to reduce latency.
1:42And this allows for fleets of agents to push in parallel. And Entire's performance is next level. Their repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market. When GitHub is down, you can still keep working, and you don't even need to migrate away from GitHub. You just sign up to Entire and the platform mirrors your repo. And here's a neat thing. Have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself oftentimes. Entire captures all the prompt history with your agent right inside of the repo, easy to check back.
2:17And it's got a pretty innovative UI to show all of this. If you're looking for Git hosting that works even when GitHub is down, head to Entire.io slash pragmatic install the CLI and mirror your repo with a click. I've already done it and I love it. Oh, and did I mention that it works with any agent and it's open source?
2:34Sam Newman: Sam, it's great to have you on the podcast. Thank you so much for having me. I'm happy to be here. Really happy to be here. Big fan of the show, as they say. I'm not going to lie. I've become a bit of a millennial and I think I watch a lot of shorts rather than watching the whole things now. I'm becoming my son. Yeah, millennial is a new boomer, apparently, I've learned. Oh, God. I was happy enough being Gen Y. Now I'm wherever I am now. Well, so let's go back to the beginning. You've been known for a lot of your books, your takes on resilience system, microservices. Before that, you worked at ThoughtWorks.
3:03But let's go into the very beginning. How did you get interested in tech and computers?
3:08Sam Newman: I'm quite fortunate. Living in the UK, we had at places an access to computers quite early on. We had an Acorn Electron at home, which was sort of one of the generations of the BBC computers that came after the BBC B. And then I got a Spectrum 128 3, which is the Spectrum that had the built-in tape drive. This is way before the Commodores. I think some of this was the BBC. I think I'm pretty sure that the Acorn Electron was pre-Commodore, but the spectrum would have been around the same time. The UK was a bit more of a spectrum and BBC market, whereas the US, the Commodores and the Ataris and things were a bit bigger.
3:46Sam Newman: So that sort of got me a bit interested. And then I always sort of stayed interested in it. Dad had a computer. I did the very boring thing. I went to uni. I did a software engineering degree at a place called De Montfort University, which the running joke in my family is it's in the top five universities in the Leicester area. there are only two universities in Leicester. I mean, it was, I loved it. Actually, it was a great class. It's basically an MIT syllabus. And then I just did the work after. I got a great, back then we did these things called sandwich courses. Sandwich course. Yeah. And so basically the idea was you would do two years of study.
4:21Sam Newman: You do a year in industry that was paid, but you effectively going in as a student. And then you did your final year. Oh, nice. Which is actually fantastic because I kind of was drifting a bit. And when I did my year in industry, so many things clicked. It meant there was much focus going in. And I got to do work for a company called, back then they were called GC Alstom. They were called Alstom. And we did software for the European Space Agency and gas and steam turbines. Yeah, it was really cool. I joined about the first time at that, and then I went back after. And when I went back, it was just around the time of the first dot-com boom.
4:57Sam Newman: And so everybody left. So I went from being the person that was doing like nothing to the person that did everything. But I remember my first week there, I was given a Fortran 77 code base. So Fortran 77 was released in the year I was born. And every single for loop had a break condition because we were basically doing analysis on these Unix machines. If the models and the time on the machines was expensive and we were worried about accident or infinite loops. This is us trying to solve the halting problem, basically. So every for loop across these 8 ,000 files had a break condition of, I think it was 99 ,999 loops.
5:38Sam Newman: And if it went over that, the program basically broke and aborted. The models had got bigger. The computer got more powerful. So it had to go through and change them more. By hand back then. By hand, well, this is one of the nicest things that anyone's ever done to me in a professional career. Because Fortran 77 is based on the old punch card limitations to an extent. so your lines can't go very long so it's all a bit fiddly you can't go and put nine and my boss said i reckon i reckon it'll take about two weeks if you do it by hand or and he slid a book across to me you can read this and he said he gave me a copy of the o'reilly said an orc book and so i spent two days learning how to do regular expressions and i wrote a refactoring using regular expressions accounting for line lengths and everything else and it's i think the closest I've ever come to feeling like a god.
6:27Sam Newman: I did it and I thought, oh, that's the thrill. That's the absolute thrill. I'm not sure if it's a joke or a story, but, you know, the two, the two builders who need to build a wall and one of them starts to build the wall one by one by one and the other one is kind of drawing, making plans and like, I don't know, like it takes three days to build the wall and the first two days, this guy does nothing and the builder building it feels really good. And then on day three, this builder like builds a machine that builds or like puts together a machine that builds the wall for them and then yeah but sometimes sometimes of course the wall falls down the moment you said the thing is i love that that story that story is all true and it's one of the earliest times i can remember learning something and having such a visual reaction and i remember this has been just towards one of the end of the thought works there's some grads in and they're asked to tell some stories about like what had happened in previous career and i told this story i said i added the nine made the models bigger and then one of the one of the grads said why didn't you just replace it all with a constant so you could just change the number once anytime in the future and i was like i thought i'm learning once again yeah so that's that's also why i love it it's like you you get humbled all the time this industry you get to learn stuff yes there's reinvention of the wheel going on but you were inventing better wheels the whole time so and i think it's just nice to remember this every now and that i'm sure everyone who's built code and programs like at some point you have the realization that you do feel like a god that you can do things with these things which is just so difficult to do by hand right like there's i think we forget about that when it's kind of day-to-day and we're just like doing our stuff or even like right now we're using lms which also do incredible magical stuff that would have been impossible but like it's kind of cool that we are in charge we can do amazing stuff you get that little fizzy dopamine here and i think that's something we're struggling with a bit now is as people are getting further away from the code they're trying to work out like how do i still get that kind of dopamine in the same way um it's not as much an issue for me because i've later on you know i've been doing this for almost 30 years now and i've already been mostly working at the area of architecture for and sort of architecture and infrastructure for a while so i feel a little bit more removed but it's uh yeah talking to my son who's a programmer he's got into it around an interesting time right now.
8:46And then you eventually made your way to ThoughtWorks where you spent a bunch of time. What kinds of things did you do at ThoughtWorks and what are things that you learned there?
8:55Sam Newman: I did anything. I mean, I joined when there were probably just under 400 people left with about 4 ,000. And I worked and got to go all over the place. I joined at a good time because a lot of the, I mean, we were very much an XP shop. And I think that the core of the business still kind of primarily is when I speak to people there. By the way, XP, was that influenced by Ken Beck's XP? Yeah, no, the extreme programming is very much, it's Ken's extreme programming. There was a bit of a London flavour. So the kind of London School of Test-Driven Development and the Extreme Tuesday Club, XTC, around here.
9:32So what's the London flavour of XP and TV?
9:35Sam Newman: It's sort of like, if you want to know, you probably need to read Growing Object-Oriented Software Guided by Tests, one of the worst titled books in existence written by a couple of friends of mine, Steve Freeman and that price. And I think that was a lot, there's a lot of people in the community around here that knew that. There's a rich meetup scene. And so I kind of got introduced to people through there and came in and I was, same time I was offered a job by somebody else. It was a bank. The bank could tell me exactly what I was going to be doing. And ThoughtWorks, basically, the recruiter at the time, is now one of my very good friends, one of my oldest friends now.
10:09Sam Newman: She said to me, I don't really know what you'll be doing. and I'm like that sounds like the job for me I went for that one and this is at a time when like you know we talk about continuous integration but the tooling wasn't there and talks about TDD and the tooling wasn't there so it was in you know it was around that sort of time that we started work I started building the build pipeline stuff in about 2004 and then I worked Dave Farley was my first tech lead Dave Farley? Yeah yeah he's my first tech lead at ThoughtWorks i was one of the people that interviewed jez humble to backfill for me on a project no way so your project aol of all places and i would use the backfill because i was like i'm sort of half his sis had been half dev really so i see myself so we we needed someone to take over my job and so i interviewed jez um and so i was around when a lot of interesting things were happening so i was fortunate enough to be there while we were sort of kicking those ideas around and obviously people like Dave and Jez put those ideas together really, really well.
11:11Sam Newman: You're sort of surrounded by everyone talking and thinking about these ideas. I think you got that. I got the idea very early on that this stuff's all up for grabs. These things aren't set in stone that we can push. And when you're saying these things, like you mentioned continuous delivery, you mentioned what other? Test automation, build pipeline optimization, infrastructure automation. I went over to the US in 2007 actually to work in a big I've actually worked at Google for like a year and a half as a ThoughtWorks employee Wow, that's quite rare We were there doing a project it's all been talked about publicly it's a great write-up by Mike Bland all about our experiences it was me, a few of me a few of my ThoughtWorks colleagues some people from Industrial Logic and some people from Pivotal the old Pivotal pre-acquisition They were known for their really excellent engineering culture Back when it was just Robby's company or anything else.
12:05Sam Newman: And we were there to teach Googlers primarily how to write automated tests and how to write code that was testable. And so I got out there and I was talking to people out in the Valley and stuff. It was like, we were going into these enterprise companies doing test automation using stuff like Selenium. And you go around talking to people and it's like people didn't, I remember having conversations with people like Yahoo and even big chunks of Google who did not think you could do functional automated testing of like websites and stuff. I guess because he just never did it, right? Yeah, absolutely.
12:35And for me, I hate it whenever I haven't pushed myself
12:40Sam Newman: or found a better way of doing something. And I was lucky enough that early days in ThoughtWorks, that was just a constant churn of ideas and reinvention. I mean, goodness knows how many mocking frameworks and inversion of control things I was. I built two different replacements for cruise control myself, right? And so, but like, you know, it was at that time I created a thing called dbDeploy. My friends Nick Ashley and Graham Tactic, which was like the first database refractoring tool and all these sorts of things, because they didn't exist. We needed them. And it was like, if you want to do it, just do it.
13:15Sam Newman: That was their mindset. And so I think there was always this idea working at ThoughtWorks was always like, what's next? And I was quite fortunate that because ThoughtWorks was growing so much, there was always things needed to be done. And my default was normally just to say yes to everything. so I got to travel all over the world it was fantastic met my wife through ThoughtWorks most of my best friends will come from ThoughtWorks now still to stay so although it's not been there for a long time it was kind of really formative I think in terms of that. And it is really interesting because ThoughtWorks has a alumni pipeline people who used to work there who are now really well known in the industry for their ideas for for thinking differently some some of educate people others uh have created businesses to to build software in better ways in certain dimensions and it's just interesting because like usually i wouldn't think of i'll just like put it in a box like you know you have product companies like like google like meta like they have they build their own products and and you have like different tiers of these companies some can afford the best people and they have an excellent engineering culture again google was known for this meta to some extent and there's some companies maybe who are like lower in tier and then you have the so-called consultancies who are, you know, they don't build their own stuff or they don't do too much of it.
14:27They come in and they help companies build or you have a problem when you hire a group of experts and ThoughtWorks technically is in this group and usually you would not think that there's too much innovation going on in these places. But with ThoughtWorks proves to me, yourself and other people, it's actually you can have, like when you are the best in what you do globally, you can actually have a lot of innovation happening there as well, right?
14:49Sam Newman: Yeah, and it was innovation with a different set of constraints. I mean, I would say there aren't that many people that have come from ThoughtWorks that were really great startup product people. I mean, a lot of, you know, a lot of ThoughtWorkers make X and scale up people once you're all together. There are actually very few real entrepreneurs that came out of the ThoughtWorks machine. I know a few of them. There aren't many of them. But I think there was always the opportunity to innovate even within a constrained enterprise environment. So much of our working conditions as a consultancy who mostly did co-source delivery at that time, we were doing kind of co-source delivery, a bit of enablement through delivery.
15:20Sam Newman: your working conditions are often based on what the client does. And unless you're there explicitly to change how the client builds software, so what you work, what you got. But that was self-perpetuating. Because we were already used to sort of challenging and pushing and trying to do new things, that spirit carried on. And so I think we grew quite aggressively, but not that aggressively, rotating people through, and that spirit kind of carried on. I mean, look, we had to start most engagements with convincing people why pairing was a good idea, let alone TDD. Yeah, and I imagine how the conversation went, like a manager going like, well, what would you pair?
15:57Then I have like, you know, like eight engineers and now you're making it four, which sounds like a terrible thing to do.
16:01Sam Newman: Yeah, yeah, which, and this is, I mean, I had good fortune because of the people, because of the work at ThoughtWorks. I got to know Martin Fowler quite well. And, you know, he supported me a lot when I wrote my first book. I still remember when he said, well, programming is not typing. I said, fundamentally, it made it click for me was having those conversations with clients. if the client thinks that programming is typing and that's why they don't want you to pair program well then you've got a fundamental problem that needs to be dealt with and it's not a conversation about pairing the conversation is what do you think programming actually is well but let's bring this a little bit forward because right now we're having the same conversation right for a lot of CEOs even founders are thinking who have been building tech companies they all look at what is the most time-consuming part of what we do in building software.
16:4950 % of the time has historically been writing the code. So a lot of people think programming is code and LMs are really good at churning out code. So what is programming?
16:58Sam Newman: Okay, there's programming. I think it's Peter Knorr's article. The guy who came up with Barker's Knorr form. Well, he's Knorr in Barker's Knorr form. He wrote his paper years ago, which he won the Turing Award for. And he sort of describes the... the program as being a theory that lives in the mind of the developers. That it's actually us together, working together collectively, who builds a program. The fact that we happen to make that program work through code, the code is really a side effect of how we work together. And I think product development is just that extended, right? What we are building is collectively, we are building a system together.
17:40Sam Newman: and the code is one of the mechanisms we use to deliver the system but it's not the only thing so i think holistically that's what it's all about it's just that we focus on the code because that's historically been where we sort of spend the time but i mean this is also a classic theory of constraint stuff though right if you speed one bit up you get constraints elsewhere and it exposes friction points elsewhere i would also argue that a lot of the time coding was not the problem or not the bottleneck it was how the coding was done delayed feedback cycles and things but i do come back to that peter law piece again because it's like margaret ann story referenced it recently in her term pay for uncooperative debt but it's like building software is a collective it's a collective endeavor it's people working together who build software uh kent put it in his 21st book that software design is an exercise in human relationships right ai doesn't change that and i think all that's really happened over say the last 15 years and my experience is that we've got a bit better at understanding that we need better working relationships like people like product owners for example but that's what the program the programming is the hour of creating is the collective hour of creating software it's not just developers that do that never has been and so after thought works you've you've gone independent right i did three startups none worked out for different reasons so don't ask me for advice on how to make a successful startup um and then i went independent yeah moved back i was living in australia at the time and moved back from australia and yeah i've been independent ever since so doing kind of a mix of stuff really ever since i'd written the first book of about microservices i thought in my spare time at thought works and then uh that book came out and i'd been at thought a long time i realized that there's always a danger of becoming institutionalized you get to a point with a company it's like i think there's things as companies do to change and i've tried changing those things and they don't want me to change them so either i'm bad at changing them which is probably true or they don't think it needs to be changed which is also probably true it's like i could hang around and get bitter i'll just go now let's go now so i went off and i thought i went independent it's been really interesting ever since because it's sort of on one level the shackles are off um the problem of working for yourself though is that the boss is frequently an asshole so yeah current Sam frequently is cursing past Sam for the commitments he's made.
20:02That's more than a lesson, I guess.
20:04Sam Newman: Yeah, yeah, yeah, yeah. I mean, as I say, I'm almost 50. I'm still not learning the lessons of that, right? And I've made those mistakes for a few years now. So I want to talk about microservices. You have written the book, Building Microservices. In 2015, you started to write it in 2014. And as reference, Uber started to talk about how they have thousands of microservices in 2015. So you were kind of writing this book right as the concept of microservices was starting to go mainstream across startups. Do you remember how and when microservices took off and where was it even coined? Was it around Martin Fowler?
20:37Sam Newman: No, it was James Lewis. So it was going back in the day. So you got to remember, again, it's like something in the water. It's like we have a bushfire. There was a lot of buildup before the bushfire kicked off. And so this would have been 2011 to 2012. James was talking to a bunch of different companies who were building what would have classed back then as service-oriented architectures. And earlier there'd been this kind of what we refer to as the guerrilla SOA movement, which is an aim to simplify service-oriented architecture, you know, shift over to REST, work done by people like... Yeah, the original service-oriented architecture had like this XML format.
21:15What was the name?
21:15Sam Newman: Oh, WSDL and Simple Object Access Protocol. never was the word simple being less well used soap soap yeah soap yeah let's start with simple and it was the most complex thing ever exactly so you had this whole heavyweight and then people like Ian Webber and Ian Robinson and Jim Webber had like been pushing centralized you had mantras like make the endpoints smart and the pipes dumb to keep things simple put all the smarts in it so this is all the background we were also at this time doing a lot of work around like I was running the continuous living practice for Europe at the time and so a lot of my engagements were like optimizing the final mile.
21:51Sam Newman: So you've written the code and it's like, how do I get it here into production as quick and effectively as possible? So basically taking the idea as a continuous delivery. So I was around when Jez and Dave started working on those in 2006. I'd had the conversations with them. So that was all happening. James was talking to companies that were creating service-oriented architectures where they could replace the services quite quickly. And so basically you'd look at it and say it's a finer grained form of SOA. So that was all happening. So James went, we had this architectural symposium up in the Lake District.
22:23Sam Newman: And James pitched this idea. He was called the micro apps at the time in this room. And somebody else in the room said, they're more like services. Don't remember who said it. There was about 10 of us in the room. Most of us were hung over. And then James and Ian, so James and Martin wrote the first post. So the first paper on Martin's website was written by James's original kind of discovery of this pattern that he called microservices. At the same time, I was doing a lot of stuff on infrastructure automation. And so I was coming at it from a different lens. So I think James was coming at it more from an architectural lens and I was coming into it from a continuous delivery lens.
22:58Sam Newman: I spent at this point almost 10 years with that being my focus. and so I was starting to see the architectural impediment so if I want to change things frequently but the things I want to change are integrated with other things like a monolith now that's how we would describe it exactly right I did a talk called Designing for Rapid Release in 2012 which is sort of touching so basically James had put a name to this stuff and put some patterns in it and I thought hang on a minute the stuff I was talking about over here these things seem quite sensible and I'd always wanted to write a book. I mean, honestly, I wanted to write an animal book.
23:36Sam Newman: I wanted to write an O 'Reilly book because that said an orc book I was given. Animal book, yeah. And so I just started writing. I've always found writing quite, I'm quite lucky that I find writing enjoyable and quite easy. So I just, yeah, sat down and wrote the book and that's it. But for me, it was an extension of continuous delivery and it was a style of architecture that was only possible because of continuous delivery and because of DevOps. we'd had these ideas before I mean Erlang is a representation of these ideas but making it real practical and attractable thing but I mean really fundamentally it's just the whole thing has been an exercise in smuggling through concepts of the 1970s which is fine by me right if I get people to pay attention to information hiding and coupling and cohesion that's alright but then as you said this was sparking before the bushfire And then the bushfire did start.
24:31Like suddenly, again, I think Uber was a very notable company starting to talk in public about having thousands of microservices on conferences like InfoQ and it got people's attention. And then it kind of seemed that overnight, a lot of startups started to say like, you know what, let's just do microservices.
24:45Sam Newman: There was a bit of rebadging going on. I mean, I was lucky when I wrote the first book that I met Ben Christensen fairly early on through the Yale conference. Ben was looking after performance and sort of reliability at Netflix. and I also knew people at REA. And these were two early... REA. REA, realestate.com.au in Australia. And these were companies that were very much pushing the envelope in different ways. And so I had two very good real-world case that I used to draw upon for my book. But most other people weren't talking as publicly around that. And also, before Netflix referred to their architecture as microservices, they just referred to it as fine-grained SOA.
25:23Sam Newman: And to an extent, you know, of the mind-standing, that's what happened at Uber and elsewhere. We didn't have a different word for it. then of course what happened was that everyone just called their soa architectures microservices which is sort of to be you know semantic diffusion is a thing is to be expected so i just kept fighting the good fight for like it's just independent deployability and you know it's i've always positioned it as it's an opinionated style of soa you may not agree with the opinions that's fine do something else then but and there are some opinions so then what is the difference between microservices and what was the other service?
25:58Service-oriented architecture. Service-oriented architecture. What are the differences? In your very strict...
26:04Sam Newman: Now, to be clear, right, I have my definition. You might have your own definition, but I'm the one that wrote the books on it. So my definition wins. That's always been my argument, right? Write your own damn book. I've always presented it as it's... There were two kind of very strong opinions I think we hold with microservices. The first and most important is that your microservices need to be independently deployable. So your unit of deployment, unit of change, needs to be a single microservice boundary. And rather than I'm deploying everything together, that's a monolithic communication. Three services at the same time, et cetera.
26:36Sam Newman: Yeah, even a mini monolith, right? It's a single service at the same time. That's clear. That's it. And the second one, which I think there are going to be exceptions to, is that your kind of default way for defining your service boundaries is around your business domain. This is taking the concepts of, you can use the management design, you don't have to, but rather than a lot of SOA architectures were dominated to an extent by technical decomposition. The classical N-tier architectures, layered architectures where you've picked your service decomposition based on the nature of tech rather than thinking about the business functionality.
27:08Sam Newman: That's a bit softer. Independent deployability is the main thing though. Okay, well, because I'm not thinking back at Uber. I was at Uber at the time when your book was starting to become really popular. I was there from 2016 and we had independently deployable services. we didn't really think too much about the businesses or domain. They had obviously their interfaces, they had contracts that they offered increasingly strongly typed and we used the different ways to do it. But it was just more like, I mean, as you can imagine, there was just devs, oftentimes not very experienced devs who were like, all right, like let's build a new service.
27:42Here's the interface, here's, and then anyone can use it, which actually like there were these things where a dev or even a not very experienced dev deploy the microservice and then others start to use it and suddenly it became, you know, years later, there would be these exercises to figure out the hot path and label services and then this like loan service was on the path of like some really, it needed to be requestable as a really high tier service because a lot of others were using it.
28:07Sam Newman: I mean, there was always, I mean, also in those sorts of environments, I was checking to people at SoundCloud and similar about that. And this is also like about corporate culture and like what was acceptable. Like in that kind of environment, just go and do stuff. It's absolutely fine. And I think that's why Uber, for example, I just talked with Twan Palm, the C2 at the time, about what was the real deal about microservices. And he said Uber, he never really wanted that many services, but it was just more of a constraint because Uber had a monolith. The story was, which he told on a podcast, we'll link it at the show notes below, that Uber had this big monolith.
28:40And it was really slow. It was slowing delivery down. And they needed to move fast because all the other competitors were out there. So they said, all right, they wanted to get rid of the monolith and put together smaller services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen. So they said the mandate was everything new needs to be your own service. You take care of it, we don't care. And suddenly so many independent services were born, which was not the goal. And years after the fact that when Uber decomposed the monolith, they're now looking at maybe sensibly merging based on business domain, et cetera.
29:12But this story wasn't really told at the time. It was just, oh, look, like Uber has thousands of microservices, almost one or two per engineer. right? And this is what the world saw.
29:20Sam Newman: I mean, it's also though, but this is the reality of microservice adoption is like, it's just a tool in a toolbox. I mean, I've described microservices as being an architecture of last resort. Really? Yeah, yeah, absolutely. Because it pushes you quite naturally to a more aggressive distribution of your state and everything else. And so you have to have a good reason. One of the best reasons to use microservices is if what you actually want is an organization with a higher degree of autonomy. If what you want is teams who are able to act independently from each other, the microservice architecture is a fantastically good fit.
29:58Sam Newman: But then really, it's a means to an end, right? So that's often what a lot of people get wrong, I think, is they adopt microservices and they think this other magic stuff's going to happen. No, no. If you want organizational autonomy, that's what you're working towards. And this is just one of the mechanisms to get you there and most of the enterprise organizations are the vast majority of ones I've worked with in this space that's what they're after they've got huge numbers of developers lots of middle management they're not getting much work done they're thinking what do we do organizational autonomy becomes you know the solution to all problems again I don't always get to have to say whether or not it's a good idea and I think that's where microservices can be especially useful I mean there are other exceptions as well and so an organization like you know, Uber.
30:40Sam Newman: I mean, and I know at Netflix as well, things like domain-driven design weren't talked about at Netflix, but the teams were already fairly well-scoped around there was a product already. And so you had those natural high-level boundaries around the business domain. So those things flowed naturally. And so that for me was a logical kind of stepping stone. You also have the difference, and this is something I realized quite early on, when you go and do the work in the dot-coms versus the enterprise organizations. When you say dot-coms, you mean the digital companies? tech companies. I'm old enough that I still talk about dog comp.
31:13Sam Newman: But the tech companies, I mean, fundamentally, algorithm, you know, from a domain concept, they're incredibly simple. The domains are very, very simple compared to enterprise organizations by and large. Domains in there are much, much more complicated because they're based on business processes that have existed for 10, 20, 30, 40, 50 years. Oh, yeah. So what you mean in a lot of tech companies, a product is pretty much like, here's our product, here's how it makes money. And that's kind of it. And we have another product. And Uber, and that Uber, like a simple thing is there's rides, which it makes money like this.
31:39And then there's Uber Eats, which is a separate business. Overtime, by the way, does get more complex. But when you start, it's like really, really simple. And it's really separate and it's clear where one ends versus where the other begins. And then you do some hacks when you need to launch Uber Eats really quickly and you build it on the right infrastructure. And then Uber Eats order has like two trips attached to it because anyway, but as you say, like early on, it starts very simple.
32:04Sam Newman: But then also in those environments, your business processes and the digital implementation of those business processes is evolving together. And a lot of enterprise organizations didn't have that. I mean, their first generation of digital products were codifications of paper-based processes. Oh, so you're talking that these companies that exist before digital, let's say, an Adidas or let's say a shoe company? Virtually all banks. Virtually all banks. And now they're like, we have all these processes and now we need to build a software that mimics what we do, which of course is... That's how you start off.
32:33Sam Newman: I mean, that's what the whole Domain Design book came out in 2000. and it was a venture design came out in 2004. And it was a reflection of the fact that the vast majority of software development was the process of us trying to model existing non-digital processes and how you do that. I wonder if we will have a different but similar problem of the same thing now that, you know, we know AI tools are here. We know that, for example, they're great at generating code. And we have two types of companies. We have a company, a digital company that has existed pre-AI. They're used to writing product programming documents, deploying developers, product managers, designers, QA.
33:11They all have their roles and it works like a really well-oiled machine. And now you have these so-called AI native startups who just, again, they have a few people. They're just, from the start, they're using agents for everything. They have a very different process. And at some point when this whole thing will mature and we will have best practices, we'll have the existing companies who, like the previous one, they will try to retrofit this tooling to how they're working today.
33:33Sam Newman: It's already happening. I mean, and the thing is enterprise companies will rarely be successful at doing those first gen things because ultimately the corporate DNA would just crush that stuff from within, right? And it'll take a while. But I think the whole industry is grappling with the idea of like, are we now moving to a new level of abstraction? Right. So software is all about abstractions, right? But, you know, we went from doing machine code to doing assembly, went from assembly to general purpose programming languages. and you're coming back to the idea of the program being the mental model, we would evolve that mental model at the code point of view.
34:09Sam Newman: That was the most effective place was us working at Java, C Sharp, whatever it is, rather than working at assembly level, right? And it's like, okay, well, it's actually what we're talking about, really. We're just dipping that level up. We're still actually building a product together. We're still thinking about what the software is. We're still building that mental model, maybe now more collaboratively involving more product people, for example, in that conversation when we're writing our specs. And so the question for me is, are we ready to make that step? And the answer is no, not everywhere.
34:42And this is a really good point because if you think about it, like one of the mantras inside of Facebook used to be that I think something along the line that code is truth or something like that. But in general, in a lot of tech companies, if you want to know how things work like where the truth lives, is it in documents or is it in the coded business logic? It's in the code in the business logic. And if it's well-maintained code, often well-commented or structured, you can actually see the business logic and the rules for, I don't know, for example.
35:12Sam Newman: See, I feel like I've got to channel my friend Charity at this point and say the truth is production. Everything else is the lies we tell ourselves. Production is truth. But I think in terms of like a definition of what the system should be doing, there is this ongoing debate about where is that definition? Is it the code or is it the spec? Brigitte Brockley did a nice article on Martin Faller's website recently about spec-driven development. And she talks about the idea of some companies being spec-first or spec-anchored. And a lot of it comes down to where they see the source of truth being.
35:47Sam Newman: So a lot of companies are using things like Kiro or SpecKit to define a feature. and then the LLM executes that spec and you can iterate on that process, right? And so a lot of people, they start off, they use the spec to write the code and then they throw the spec away. And then you get other companies that keep the spec and then when the feature changes, they just update the spec. And so they're sort of seeing a bit the code and a bit the spec as a source of truth in terms of what the system should do. And then you're now starting to get companies where they didn't even ever look at the code and they just look at the spec.
36:23Yeah.
36:23Sam Newman: And so we're kind of going on this spectrum. She had some great terms for these. And I think for me, it's like, is that the right path? I don't know. I think there's a lot of questions about what we would need to do to get that. We have a degree of confidence about what happens when I execute a line of Java code. Beyond the safety of the compiler and the runtime and all that stuff, there's a lot more questions to answer, I think, before the whole world is ready to say the spec is the source of truth of what our system is supposed to do. production is always going to be the real truth so just to be clear but but i think it's a good question to just ask yourself what however you're building however you're using agents of where is the truth of your system right now is it more in the code do you have a spec is it scattered between the minds of people uh which goes back to the the collaboration because again a lot of the reality of a lot of startups are is a few people hold these things in their head right which is okay in the beginning but of course as you scale one of the reasons we used to love code review which is now going out of fashion, but we used to share information.
37:25And when someone left the company or got ill or, you know, the famous bus factor, it was okay because other people understood because they reviewed, they were taking part in that part of it. But now we're with AI code review is some, some, some companies are like, well, we're not going to do it because of it's too much code.
37:41Sam Newman: That was always one of the superpowers of pairing, right? Pairing. You wrote, if you do, if you do promiscuous pairing, you rotate every day. When someone leaves, everyone's got the same shared context. That was something people missed around one of the benefits of pairing. I think there's a lot to be said around this. I was chatting to Chris Ford. He's sort of head of software, head of technology for ThoughtWorks in Spain. And he said the biggest problem we've got around AI-driven software delivery is tacit knowledge. There's a lot of knowledge in our heads that we've built up that we don't necessarily know that we have.
38:17when you use something like a code review you probably had looks at that and said that doesn't
38:21Sam Newman: really feel right i wouldn't do it like that and sometimes it's subjectivity but sometimes there's something about that that you can't quite pinpoint maybe you can unpack it maybe you can't but it's also like you have bunches of things like that in your head about how software delivery should be done about how the business works everything else if we want to step up that level of abstraction to work at the spec level, we actually need as human beings to be able to unpack that tacit knowledge that we have and make sure it's in the spec if we really want to be hands off. I also think there's a fundamental problem that we've got in the current way a lot of people are doing software deliveries.
38:57Sam Newman: You've got, and this comes back to the cognitive debt stuff, which is you've got a lot of people running off in different directions. I've seen a few people talking about pairing with their AI assistant, not a human. Stop pretending it is. And so you're all going off, you're doing more in a different direction. That shared mental model of the program is breaking down and there's gaps in between. AI was supposed to free us from drudgery, right? It isn't for most software developers because we're doing more work, we've got more context switching, we're losing that big picture. So I think if we can work collectively on the specs, we can rebuild that model, then that absolutely could be the source of truth.
39:32Sam Newman: I still think it's going to be parts of our brains and part of the spec, right? But for that to work, there does have to be the collaboration piece. Whether or not that's our dev teams go from being 15 people all in to about five people or six people all in, don't know. But collaboration is really important. Chain Guard are a proper company, right? They do proper stuff. They're excellent for them. They recently announced they were getting rid of any human-based code reviews and they said we're going to spend more time working on things like effectively working at design doc level. And actually I'm okay with that because firstly I trust that they know what they're doing.
40:06But also there's something about, I mean, I think a lot of companies are making the mistake of not doing the thinking.
40:13Sam Newman: Whereas Chengar is saying we're going to spend that time doing more important thinking. Right. And I think that's sort of how we should view this. I'm OK, you know, not looking at the code as long as you're doing other things to make sure the system is operating correctly and that you're thinking critically about how the system needs to change and evolve. if you're just not looking at the code and you're not caring about those other things well then you deserve everything that's going to happen to you which also goes back a little bit to what Charity was talking about on how you want to have systems that value that works about having empathy for example for the work that QAs and SRE folks have had to own code in production that they did not write they don't necessarily read or even understand but they know how to operate, they know what good looks like in production, they know what what it looks like when it goes poorly.
41:04Sam Newman: I mean, that's also a bigger thing that now more developers are having to struggle with because if you want to do spectrum development, you have to know what good looks like. I run these sort of half-day conferences for O 'Reilly online. We did one recently on software delivery. A load of speakers and about six speakers and four of them were talking specifically around different varieties of spectrum development effectively. And all of them came back and said, You can't do this unless you can define what good looks like. And if you can't, don't bother. And, you know, some of those people were much more bullish, like Addy Osamai is very bullish on all this stuff.
41:40Sam Newman: But one of the presenters who was doing, they've got like a small software fracture, only 15 or three people, but like they're actively pulling in their observability data, their O-Tel stuff, right, into their agent to understand how it's changing. And I think that's exposing, I think, more developers into understanding. I mean, this comes back to the acceptance criteria stuff people do in stories. They're always very narrow and functional requirement stuff. So now it's like, I think we'd have to be more grown up about those things. The discipline is absolutely important and the expertise is still needed.
42:11Sam Newman: It's just needed at a different place. Sam made the really good point that you cannot hand your work over to agents unless it does these two things. One, define what good looks like. Two, prove that your system does what this good definition looks like. You already know who I need to mention when it comes to proving that your system works as expected. Antisysys, our season sponsor. Agents are writing more of the code, so how can we trust that our system works correctly without bugs? Code review is an increasingly naive and ineffective approach to do so, which is why Antisysys goes well beyond this practice.
42:43Antisysys runs your whole system in a hostile simulation. This simulation includes both targeted testing and fast testing. By running this simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it a lot easier to fix issues. The first thing I thought when I heard about Antithesis and that it's automated bug discovery and fully deterministic testing, that it sounds like science fiction. But actually, it's hardcore engineering when you look under the hood. JaneStreet, Fly.io, and the etcd community ship agent-written code with full confidence because they know it's been verified by Antithesis.
43:20To see more case studies and details, head to antithesis.com slash pragmatic. I'd also like to mention our presenting sponsor, TurboBuffer. We know it's a database that's ridiculously fast and extremely cheap. And they are doing something amazing. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. If you follow TurboBuffer, you know that their storage architecture was a massive part of their early success. Redesigning a winning architecture is a big deal. It's one thing for query plans to pass unit tests which is already happening.
43:52It's another to bring each query plan to performance parity or better while maintaining correctness and reliability in production. And the team is sharing the messy reality of the rewrite and how the v3 version is currently a lot slower than the current production one. For example, we can see how cold full-text search is 11 times slower than prod and hot full-text search is 126 times slower. But we can already see some wins like hot and cold vector search is at parity or better. I really appreciate this work in progress view and the real-time logs and what it takes to do a successful storage architecture rewrite.
44:24The TurboPuffer team is keeping on the work log of their journey and you can follow along at turbopuffer.com slash v3. I definitely will do. And with this, let's get back to Sam and what QA and S3 folks already know about running code that they didn't write. So I want to talk to you about your latest book that is just coming out. building resilient distributed systems.
44:44Sam Newman: Yeah. What is the book about and why did you feel this is a good time to write it? Years and years and years ago, I went to Velocity, which was quite a well-known conference. It's a conference. It's still on? No, no. It was the O 'Reilly conference and it was like the conference where a lot of the early dot-com people went to talk about how they made their systems scalable and stuff. I got to go. It had been like one of those career ambitions to go to one. I got to go. I was after I'd written the first book and I got chatting to John Allspoel there, who I still chat to for now and then, he now works in this resilience engineering space, pointed me, I don't know if it was him or someone else, but it pointed me at this paper by David Woods that was all about resilience engineering.
45:21Sam Newman: And the paper was utterly baffling to me, in absolutely no sense to me. But it was all, because it's not, yeah, because it's just, it's not written for computer scientists to read. And I'm not even a computer scientist, so I was really in trouble. But it was really fascinating seeing this whole other world about how we make, how people view resiliency and then what we do in the software world. And people like John were trying to take some of those ideas in a little bit, but I find it very difficult to take some of those ideas from sort of safety critical systems and that type of thinking and make in any way, shape or form palatable for normal people like me.
45:55Sam Newman: And so I kind of, this has been back, what, 2016, maybe, 2015. And so I'd go back and try and reread that paper every six months or so. And I'd read bits around the edges and see other smarter people than me that summarized it, data better. And then I was thinking, okay, these ideas now are starting to make sense. And then also I kind of realized that despite my best efforts, lots of people seem to be building microservice architectures, even though if you've read my books, most of my book is about telling you not to do it. But you have to read the book, which is called microservices. Let's be clear, as long as they buy the book, that's the most important thing, whether they read it or not.
46:30Sam Newman: And so then I realized that there are more people exposed to the horror, pain and suffering of a distributed system. And so it was like, okay, so I've got this stuff happening. I've got this, like my, you know, my son's developed now. And I almost thought that if somebody found themselves building a distributed system, working as part of building a distributed system, what was like the kind of equipment of a rubber ring I could give them, say, stay afloat and don't drown. And so the goal of the book is to start you off with really easy, foundational, simple kind of engineering concepts about how you can start making, dealing with the problems of distributed system but to also take you on a journey into kind of the whole concept of resilience engineering.
47:11Sam Newman: And so it starts off with really easy ideas and then you study it gets more and more complicated. And, you know, so I introduce kind of concepts like, well, a socio-technical system is, the difference between safety management and resilience engineering. These are all like big concepts. But the second chapter after the introduction is just like what's observability, let's talk about OTEL. The third chapter is what's a timeout, right? And these are all, I think, quite tractable things to talk to people about. They're building blocks, right? Yeah. The primitives for these systems and the vocabulary as well.
47:41Sam Newman: Yeah, absolutely. And to give, you know, to go through and sort of distill some of that knowledge down. And also to be a bit of, I'm quite happy being a bit opinionated on these things. But I'm also lucky now because partly because of my early work, I get to chat to more people. So I can actually talk to people who this is their full-time job and try and distill down some of that knowledge a bit. And so I wanted that book partly as about being evergreen. but it was also about saying there are lots of people building a distributed system let's put what type of distributed system you've got off to one side for a moment there are lots of people building them whether they should or not and they're really important to us now so how do I help people start and grow and that's what I wanted to do I mean I enjoy the process of writing anyway but this is a book I wanted to be able to give to people that's always been why I wrote a book I want the books of that book every now and then I've gone sat down to write a book and I found someone else has done it already it's like well that exists now so this for me was a book that i wanted to exist but didn't that's kind of why i wrote it yeah and i think it's a book i i certainly could have used like it got dropped into a place like goober where we did have what i realized were distributed systems you don't just large systems handling a bunch of stuff lots of services depending on one another because even a simple thing of like knowing like oh you should pay attention to observability that i mean you figure it out after a while yeah but it takes a while yeah and i think that's i mean again i mean maybe this is this is my attempt to unpack some of that tacit knowledge to an extent i've made so many mistakes and i keep making mistakes i think with making mistakes is useful but i think also creating opportunities where people can learn from other people's mistakes first is a good idea too and so in the book i also try and pull in like real world examples and sometimes from big companies sometimes from my mistakes to kind of make this stuff seem real and i think the other thing i I think I really try and get across is that things go wrong.
49:30Right.
49:31Sam Newman: I mean, engineering is all about making sure as many things go right as possible. It's not about making sure everything goes right. And there's also the only way you can build resilient systems in general is also by being honest about mistakes that have been made and talking about them and sharing them because that's how everyone gets better. And if you don't do that, you can't build resilient system in your company. And so this, again, is my attempt to sort of highlight all the awesome work that's gone before, the fantastic post-incident reports of people that some companies share. Like when there's a system that we all use or many of us use has an outage.
50:07I mean, it's annoying, especially when you're using it. But I always greatly appreciate when the companies, because they don't have to do this, but when they share a public post-mortem or incident recap, whatever that is, they go through what happened. For example, GitHub had an outage and they share the really interesting one on what all the things that went wrong and in what order and then you also have companies where something like this happens and they don't do anything so of course you're not obliged to in any way but i always have a bit of a lesser of they are obliged
50:37Sam Newman: to now a lot of them uh regulations are now making them which i which i think is great yeah things like dora and stuff in the uk in europe rather and i think there's also we should also separate around that post-incident report stuff, that there's marketing and PR versus have you done a proper post-incident review? Now, I do think you've got a duty of care really to let users know it's part of a proper post-incident review, but those companies that maybe say nothing, they might do a fantastic process internally. There's also the fact that some companies that put those things out, you read them and you're like, this does feel more marketing.
51:07Sam Newman: I think a great, if you want a good, a company that puts out some of the best self-flagellating post-incident review write-ups, go for Cloudflare stuff, They absolutely go for themselves in those. And it's like, I feel like, guys, go easy on yourselves. Everyone makes some mistakes, but I quite like. And they did it rapidly within 24 hours. And they're a publicly traded company. Yeah. Which I think just goes back to the myth, like some of them are like, we're publicly traded. We don't want to do it because it would give away information for shareholders. No, like Cloudflare does it and they're fine.
51:39Sam Newman: Exactly. And also it's. And it builds up trust. It is. I mean, a lot of how an organization is perceived is down to how you deal with mistakes when they occur, right? I mean, nothing ever goes perfectly for every company. So in your book, you talk about the three rules of distributed systems. What are these three rules? So I really tried to distill this down because I could never remember the eight, the rule of eight that came out of Sun. The fallacies distributed computing. I always forget what those eight were. There's loads of complicated stuff, but I realize when you strip it all down, there's really only three things that really cause all the other problems.
52:14Sam Newman: The first is that you can't send information instantaneously from point A to point B. It's physics. I mean, now it turns out when you get into it and start talking to people about quantum physics, they say, aha. And there's a thing called quantum entanglement, which does allow for the theoretical instantaneous read of electron spin states across arbitrary distances. at which point I point out that A, that only works for reads as far as I'm aware and B, not a networking protocol. So like, but it takes time, right? So I'm going to get from A to B and that, I can't control all of that. I can control bits of that as a developer.
52:49Sam Newman: I could control how much data I send. I could control the payload size, technology I use to serialize and deserialize. I'm unlikely to be specifying the network cabling and the equipment we use or distances between them. So not really under my control. and the moment you go over the public internet, BGP has a bad day and your packet has gone down the wrong trouser leg of the internet, right? Yep. So that's the rule one. It takes time. And that's important to recognize because sometimes it looks instantaneous. It's not. Second thing, sometimes the thing you want to talk to isn't there. Sometimes that's under...
53:21Sam Newman: And we do things to make it under our control, right? Like we would run, say, multiple copies of a service instance behind a load balancer. What happens is a load balancer fails. But I mean, like, you can't control for everything. I've had services taken out by a fire in a data center, a SAN caught fire, and specifically one bit of a SAN. The SAN had a caching storage layer with SSDs, these two pairs of SSDs. One of the pairs of SSDs caught fire. The other pair, the SSDs, were working fine. And so we had these weird latency graphs coming out of the SAN, like perfectly fine if your load went through the ones that weren't on fire.
53:57Sam Newman: But the failovers in the other one were routing. To be fair, it was routing around the bits that were on fire. I had a system where we had ducting between the office buildings and rabbits moved into the ducts and ate the networking cable. I can't control for rabbits, right? And sure, these things are like, they're real. These are true things, right? Yes, you can control for that and you can reduce the chances, but you can't eliminate those chances, right? So sometimes, rule number two is sometimes the thing you want to talk to isn't there. And the third rule, I'm very grateful to an old colleague of mine, Ben Butler-Cole, for saying I'm missing, is that resource pools are not infinite.
54:31Sam Newman: meaning you can run out of computing resources cpu memory i o whatever basically your load balancer could be overloaded and then just not respond even though it's there and we know the latency but it maybe a skew fills up and look the vast majority of the time when you have a system outage it's down to that some resource has been totally saturated somewhere and it might have been that saturation might in turn have been caused by the fact that something is down and you're in a retry storm which is in turn causing you know connection starvation which is in course got resource saturation which is in turn yada yada yada yeah but that that's the killer and and again i think we understand that you know sort of intellectually all of these things but you can go to all the concept complicated concepts and distributed systems and so why is bias and time consensus protocol a thing well let's just go all the way down to this bit what about cap and pack Okay, let's just put ourselves on the side.
55:27Sam Newman: Remember these three rules first, and then most problems end up becoming a little bit easier to deal with. And in the book, after you introduce these rules, you start, as we mentioned, the second chapter is observability. To me, that kind of suggests that this is a really, really important part for distributed systems. Why is observability so important? I mean, the way to think about observability is it's a property of a system, right? So the more observable something is, the easier it is for me to understand how that thing is behaving based on the external signals. When you run a system, which is a single program, you actually get some pretty high-quality signals coming out of that program, therefore give you a high degree of observability.
56:07Sam Newman: I can get a lot of information out of that. I could attach a debugger to a process. I could generate a core dump and run that in. I could do a thread dump on the JVM and work out where my semaphores are blocked. If this is my whole system, like knowing the cpu is stuck at 100 is actually really useful information because that's probably a bad that's a problem right So this basic information that we pretty much get for free with a single program gives us a high degree of observability. It's a single program running on your machine that you have access to. Exactly right. But now we go to a distributed system, you don't have access to the computer in the same way.
56:39Sam Newman: You've got multiple computers. And so those basic signals are harder to get. You can't correlate them unless you do extra work. And those individual signals by themselves don't really give you a proper sense of how the system is behaving. So you have to do more work to make the system observable. And as annoying with these sorts of things, it's much easier to do that from the beginning than it is to retrofit it. And luckily we've got things like OTEL now, the Open Telemetry Protocol and specification, because that at least gives us an open framework. A vendor agnostic. Although Honeycomb are the best.
57:15Sam Newman: Charity pays me in stickers, just to be really clear. I always think, I work independently, I have to be very clear, charity pays me in stickers. but like we get a vendor agnostic open standard for this now so it makes it so much easier for me to have the conversations about this but if i don't get the signals coming out of the system if i don't get the that stream of structured events telling me what the hell's going on with my system i'm flying blind like how do i know what appropriate timeout is for my client point of view talking to a service well i'm only going to know that if i have a sense of well what is its normal response time and distribution so i'm going to need latencies know the success or failure those latencies, probably want to be outplot on a histogram maybe to take a look at it.
57:55Sam Newman: If I don't have that information coming out, I'm not going to have to specify timeout correctly. And then we're talking about kind of low-level thing like latency and things that we can measure at the machine level but of course, is this observability when I'm talking about like, does it work for the user or the customer? Does it do what they do when they click the button to submit? Does it submit? Do you look at that because we know observability is usually we talk about these kind of latencies, the statuses, the 500 responses that's kind of bad, that means it's down but there's always a player that I always wondered and I found it hard even when we're working in teams to okay like we have the easy observability like this is kind of text not textbook but it's the metrics it's the metrics and the logs but then there's the stuff of does it work does it do what people care about like again in the case of like an Uber like when people hit a button to request a ride like does it actually request within usually five seconds or something like that or is it stuck in an infinite spinner, which is bad.
58:51Your locks can be as healthy as it can be, but people are upset, right?
58:55Sam Newman: You treat the raw data of making the system observable as a stream of structured events, and then from that you build up the higher level abstractions and of use, right? So use these as primitives. Use them as primitives. So a trace, right, where we get a sense of, for a given click of a button, what happened in the production system. And this is a big blind spot, right? Most organizations had to build their own tracing systems. So this is where things like the open tracing stuff came from and things like Zipkin and things like that were built as an attempt to recreate. I mean, most people were just trying to recreate what Google had already built there.
59:28Sam Newman: And this was an attempt to say, okay, can we at least look at a flow of calls? What happened during that flow of calls? Where did these things go? Well, that requires me to get a bunch of spans across from all these different machines. And a span meaning? A span meaning an activity, basically. It's something I did. And then a span can begat more spans. so I can say my first span might be a call has hit my server. So I've now got a span that starts. Within that span, I might talk to my database and then do some other operations and I can put additional information on those spans. And then I can go up to another service and then you get a trace ID that's shared across those spans and you've got timing.
1:00:03Sam Newman: So now you get effectively a view of what's happened across those calls. But that's all just based on individual events that we've been stitched together. And then you also start getting into the situation which is, is my system behaving correctly? and you know this is going away from again as i think half the time i'm just saying what charity would say um you know again going away from the vanity dashboards this is moving into the sort of the world we call the slo's right service level objectives and slis and sli so an sli service level indicator that's our way of saying at this given point in time for this thing is this thing good or is this thing bad and we use a collection of those so one example of that might be a successful percentile response time, right?
1:00:44Sam Newman: So that would be, is it a good one or a bad one? It could be a bad one because it's an error. It could be a bad one because it's slow, good or bad. And I collect those together and that helps me know if I'm meeting my SLO. So I might have an overarching SLO, which is I should be able to request a ride, the 99th percentile successful ride request within 300 minutes. And that would be your statement and that would be a commitment. The SLO is supposed to be a commitment. That's something our team is working towards. You then get this raw data. You calculate your SLIs on the individual things, and then you use that to work out what your SLO is looking like.
1:01:21Sam Newman: If it's green, everything's at a good time. If it's red, people are unhappy. I mean, you get a bit of red. There's way more nuance to it. That, I'm taking a very bad introduction to this stuff, I think. But without those building blocks, because I talk about in the first chapter of the book, nobody says, I don't want my system to be resilient. Yeah, no, no one. But at the same time, resiliency is not a binary property. It's not like off or on. It is a bit like observability. It's a degree. So then the question is, how much do you need? Because am I having a conversation with you about multi-cloud and, you know, doing complete rewrites of systems for critical standings like Monzo do, for example?
1:01:55Yeah, they have a standing bank. They have a whole standing bank. So it's sitting there, it's costing them money for the most part, but then just a few weeks ago, their main application went down for a short period of time and they activated that bank, which I'm assuming it costs them millions of pounds or dollars per year just to keep it there.
1:02:13Sam Newman: It's not even a slight sidebar, but so firstly, their standby bank has a real stripped down list of features. It's much more bare bones, but it's all the core ones. Like you can freeze cards, you can take money out, you can check your balance. It's a complete rewrite. So it's no shared code because they're worried about a bug. different cloud vendor it's also by design significantly more cheap to run so they've got thousands of microservices in the full stack this is about a tenth the cost of the cloud cost they also run it constantly yeah so that is always there um but like monzo know their monzo right that's how that works but like there's no point you starting there so actually the reason i put the observatory stuff in is like you need the raw data but also you need to make decisions about, comes back to what good looks like.
1:02:56Sam Newman: Then you need to know, okay, do I need to worry about multi-cloud, vendor, VR strategies, because that's where I need to be going? Or am I somewhere else? And so I think without that kind of sense of your North Star, how resilient do we need to be? Without that defined, you don't know how deep you need to get. This is why I try and say, how secure does my system need to be? Have you done a threat model? Have you done a threat model? I don't know. And I guess to answer this question of how resilient do I need to be, it greatly helps. And maybe it's actually a requirement to do an informed decision is how much will it cost me in terms of operating costs, in terms of time to build it, in terms of, for example, latency.
1:03:36Just a simple example is if you say like, OK, you know, the business is like, oh, would you like multi-cloud? And business is like, yeah, we'd love multi-cloud. Like if Amazon goes down because we heard they had an outage last week or a few months ago, I actually want to be able to run Google Cloud. But then you'll be like, OK, well, first of all, this will mean we're paying two cloud bills. And if we want full replication, it might increase our latency because we need to wait for it to write for both of them, which might mean slower, you know, customers might see things slower. Anyway, there's all of these trade-offs.
1:04:05And in my experience, like oftentimes, like you might retrofit that, but if you can have a discussion upfront, that's great. But to have this discussion, you go back to the basics, you need to know what the trade-offs are. You need to know what's happening in all these dimensions that, you know, you care about, like, how long will it take to build? How complex will it be? How much money will it cost to operate?
1:04:26Sam Newman: But also as technologists, we can't just say, oh, we have also got to get good at communicating with our business stakeholders, with our end users. Oh, yeah. What percentile latency do you want? Like, some developers know what I say when I say that. Not all, but some. how many you know experts in their field who are great product owners or subject matter experts but are not software developers would know what i meant by that and so we also have to understand the business domain the product domain well enough to do the interpretation and help and to bridge that gap and it is a dynamic thing as well i mean this is the whole thing about resiliency in general right a resilient system is one that changes it changes and it evolves if you don't change it evolved you're a britain system you're not a resilient system and so even those targets we set ourselves that conversation we have to have about what is good enough does change but like think about something like in the case of say the us east one outage towards the end of last year um there were lots of on aws on aws as a aws us east one region went down it was the oldest biggest region big region lots of people there first because that's often where new services got launched historically when that went down it caused a lot of people to say do we need to be multi-region and some of those companies would have had the conversation before and decided no we don't and then they'd seen the fallout and they may have changed their minds and that is fine because this is what you mean by a resilient system evolves yeah you thanks the real world but also like it is about what problems you're not going to try and solve there were some big name companies that got taken out when us east one went down and got severely had severe degradation in terms of what they could offer yeah companies like zoom for example they weren't taken off entirely but they were severely affected do we think that people working at zoom are idiots no do we think they knew that if a cloud region went down they'd be impacted yes they probably looked at that and said well for us to come out with a multi-region approach it's going to have this impact is going to cost us this much money, it's not worth it.
1:06:28Sam Newman: Let alone talking multi-cloud. Because multi-cloud versus multi-reach is a whole different conversation. And that's okay. And I think as long as you've had the conversation, and as long as, this is the other secret bit of it, of course, is as long as what you think you need to do is in tune with what your users think you need to be doing. This is why I make the point in the book when talking about the difference between SLAs and SLOs. The SLA is what you're going to get sued for if you breach. if you only ever deliver to the SLA your customers are rarely ever happy you've got to do more than that right so there is the social contract there as well now with building distributed system building resilient systems it really helps to understand concepts that are not as straightforward and I'm not going to go through all the things you explain a lot of things in the book but I do want to touch on a concept that I found a bit tricky to understand and then appreciate and you really appreciate in distributed systems which is idempotency yeah what is idempotency So there's a mathematical concept behind idempotency that I don't understand.
1:07:25Sam Newman: But basically, an idempotent operation is one that you can, you know, you can apply that operation multiple times without it actually having an effect multiple times. So it has an effect once, first time. A really good example of an idempotent operation would be pressing a lift button. So you want to go down in the lift. You press the down button, it lights up. If I press that down button again, nothing's changed. the lift is still coming. Well, we hope the lift is still coming, right? That's an idempotent operation. Some light switches are actually toggles. I toggle it, light goes on. Toggle it, light goes off.
1:08:01Sam Newman: That's not an idempotent operation. In computing, we have some operations which are not the idempotent reads. A delete is typically idempotent, right? But the example I think I use in my introducer's concept is one of payment. I was about to say that I have a lot better idempotent example for you. Yeah, payment is a good one. So I'm going to pay you money. So I'm a service that wants to issue a payment and I send you, please pay Sam£100. I'm NLPing people back at home. Please say pay Sam£100. And I don't get a confirmation back. I don't get back a response. The problem is from my point of view, there are two possibilities.
1:08:41The first possibility is that you receive that payment
1:08:44Sam Newman: and you've done the payment. It's been completed and the result has got lost to me. for some reason I didn't get the response. That's one possibility. The other possibility is that you never got that. You never carried out the operation. Maybe it didn't get to you. Maybe you crashed halfway through. I think like a good example that I always, I've seen it in the past and maybe it's over, but airlines, you're about to buy this airline ticket for$1 ,000,$2 ,000. You've checked all of your details and it says like play or reserve. And then often some airlines would take you to an intermediate page which says like, please do not refresh this page.
1:09:19or submit it again because you might be double charged. And what you really want is you pay, you know, like pay. And if you get the spinner and it's not idempotent if you do it again and you get charged twice. Yeah. But it is idempotent if you refresh, you copy the same thing, you submit the exact same form.
1:09:39Sam Newman: Oh, that's interesting. So then the question then becomes, right? So I sent you that payment, put my airline ticket, whatever it is, and I haven't had that response back. But I really want to make sure that the airline ticket booking goes through. This is you with the browser or the people at home sending Sam 100 pounds. You really want to make sure Sam gets his 100 pounds, right? Now, when something doesn't work in a distributed system because of rule number two. Rule number two meaning the thing might not be there. Rule number one, sometimes it can take ages. And accepting this thing is, you just try it again.
1:10:09Sam Newman: You retry. Because you really want it to happen. And in retrying, there's the danger that if the first operation had been processed successfully and you didn't know about it, or the first operation had been applied successfully, and you do it again, you're not getting paid twice and it were two tickets. So that's the problem with an idempotency. You want to make it safe to retry important operations. So to do that, we need to make them idempotent. There's two ways to do it, really. One that's good, and it's the one you should do, but you kind of have to retrofitting is paying. and one which is the one you just explained, which is useful, but has problems, right?
1:10:44Sam Newman: The first one is item potency keys. So you generate, you have a unique ID. You'd normally use a UUID for this. And so when you send... The unique ID, which you generate, and when you generate two of them, they should be different. There's like this tiny, tiny, tiny, tiny, tiny, tiny likelihood of clash, which again, what happened, I think. I know, but I think with UUID 7, I think it's like a 50 nanosecond chance. but it's something you can generate locally without an additional roundtrip. Yes. Your collision chance is extremely low and there's ways to modify that. You can work with that for the most part.
1:11:17And this is - Unless you're Amazon probably.
1:11:19Sam Newman: Well, this, but this is, I mean, the AWS APIs, you see Stripe, Adium, PayPal, they all use the same key scheme. So you send this to your UID. So then if you send that to me, you say, pay Sam, and here's the idempotency key. I don't send you back a response for whatever reason. You retry it, you pass the idempotency key, if I had carried out the processing, I could look at it and say, oh, have I processed this? Yes, I have. Okay, great. If I have processed it, even if I get a second or a third or a fourth, have you processed this? It's processed. I just keep going, yeah, it's processed. It's processed.
1:11:50Sam Newman: It's fine. That's a nice, simple thing to do. The issue with making things idempotency key is it requires both the client and server to support it. So it has to be a required field to go through and it has to be sent to server support. So if you're trying to retrofit this in, it's a pain. But if you're building it, use an item on ZQ. Exactly. If anything needs to be item potent, do that from the beginning. The one you sort of mentioned was, I go back, I hit refresh, I fill in all the same fields again, and I submit the airline booking again. You should know it's a duplicate. That's another one you can do.
1:12:22Sam Newman: This is good if you want to retrofit item potency in. So if I've sent the same fields, so the same person, Sam, the same amount, 100 pounds. Just remember 100 pounds. And I'm sending this through again. What you can do on the service side is generate with a fingerprint. So I take these fields, I then generate a hash that represents those fields. So if you do a second booking, it looks the same. And you're using that as a retrofitted item potency key. Effectively, yeah. There's a couple of downsides with that. You do get false negatives with that. The false negative meaning that? It might get legitimate second requests and get rejected.
1:13:01Got it.
1:13:01Sam Newman: So a real-world example, this happens in, I know this happens in Denmark. It might happen elsewhere. I was speaking to people in Denmark who works on a payment system. You go in, you buy something for 40 kroner with this particular payment system and give the car, scan the car, 40 kroner, whatever. If within the next sort of five minutes, you buy something else for 40 kroner in the same shop, the till, the handset rejects it. If it was 41 kroner, it'd be fine. If it was 39 kroner, it'd be fine. But that's because the chances of you buying something for the exact same money is so remote. They're doing some basic item potency to top double taps.
1:13:32Sam Newman: And so when you get situations where operations can legitimately look the same, but actually be different and carried out at the same time. So a really good example would be the AWS APIs. It is completely legitimate for me in a for loop to provision, say, 20 EC2 instances. Now, there's better ways of doing it than sticking in a for loop, but I could. So I could iterate through the for loop and spin up 20 identical EC2 instances. If I just relied on there being a fingerprint, they'd all look the same. so that's where this stuff breaks down um and so if you've got situations where legitimately you have operations that come that look the same but they're actually different look a bit and close to each other you have an issue if you've got a bit of space around them what you can do is hold on to those fingerprints for a period of time five minutes ten minutes an hour so there's still some potentials you can still let you because it's not ideal item potency key is the way to go So fingerprints, when you have to retrofit, if you use them both together, that can be really useful.
1:14:34Sam Newman: So you can pick up situations where people change the parameters between the retry attempts, which leads systems into a confusing. It's a bit more of a complex topic, but a lot of public-facing APIs will still generate a fingerprint. And so if you send, carry out this operation and you give the same item-ponency key as you gave before, but you've changed the parameters, the server can reject it as being effectively an invalid request. So that's how those things work together. The ideas are really simple, but there's some nuances about implementing them. There's some interesting stuff that crops up in terms of how you store them on the server side and things.
1:15:09Sam Newman: But the fundamental ideas are pretty straightforward, I think. Another interesting topic that comes up oftentimes, these days it feels more often, is thundering guards. Yes. It's when a system restarts and suddenly it faces this big load. Specifically, GitHub had an outage now a few weeks ago where it was networking related and recovery took a lot longer time because there were retries going on and it created the thundering herd. What is a thundering herd? Genetically, I talk about it in any situation where you've got mismatched capacities. You've got too many requests for too few resources, right?
1:15:42Sam Newman: Resource pool's not infinite. Coming back to rule number three. Yeah, rule number three. Yeah, so in the chapter, I talk about a whole bunch of different ones from different types of scenarios, right? So you've got everything from the malicious thundering herd, like a denial of service attack, right? A distributed knowledge service attack, malicious attack, often short notice, and there's not that much you can do about them. The answer is use fast or Cloudflare, but I go into details about how that happens. You get things like kind of internal self-inflicted ones. So a good example would be something like a cache collapse.
1:16:14Sam Newman: A system relies on the cache not only for speed of operation, but also for reducing load on other components. Cache gets restarted. you then have a lot of cash misses you're hitting the origin and the origin goes down the origin was not expecting such load yeah so a common thing there would be you first you you'd flush those cash snapshots to disk reduces the chance of you being out or having high cash misses over a restart but also a lot of companies i know will just turn their services off until the cash refills and so that's that's kind of another example of one um retry storms another one back in 2017, Square had a big outage.
1:16:52Sam Newman: They'd restarted Multipass, which was their multifactal authentication system. Multipass, in turn, pulled its data from Redis that they were using as a database because you can do that, apparently. I don't think it's a database. Well, they were using it as a cache, basically, right? Well, no, I think they were using it as a database. There are some people that think Redis is a database. This is a different podcast. I'm thinking of starting a stupid technology podcast where we talk about using technology in a stupid way. So that's a separate conversation. Anyway, so they were putting this data back from Redis and they had a problem that the clients in multipass were basically, if the Redis calls failed, they'd try again, but without any delay between the retries and they had a hard-coded limit of 500 retries.
1:17:29Sam Newman: So they just basically smashed Redis and they kept restarting Redis and then they'd restart multipass. And again, this is a long time ago. And they fixed it, right? They fixed it by having a more sensible retry policy and externalizing those values. That's another thundering herd. The last kind of sections I talked about were ones caused by like products being too successful, like Twitter and healthcare.gov, right? This is where you have an external, real legitimate load that takes you out. And the thing is you look at all of these as solutions, although what's happened is the same, the kind of solutions are a bit different.
1:18:03Sam Newman: So like when I've got a thundering herd that's coming from externally that's legitimate, well, I've got different things I'll look at. Like I will have more, I'll have a more constrained signup. I'll have fewer invite codes. So with healthcare.gov, one of the issues, there was many others, but was that they went live in like 36 states at once. And they were massively overloaded. So in hindsight, maybe going live in one state would have been a good idea first. For example, with Twitter, they were going down all the time. So they could have constrained signups or put tweet limits on you and things like that.
1:18:34Sam Newman: Those are things, and they did those things as well, right? And so although they look the same, the mitigations for Eton was a little bit different. so I kind of quite like talking about that because they were all great stories I found and of course I think we're talking about solving it with when you have the same resources but then we get into things like scaling like can you scale horizontally, add more resources because there's two sides of this right and so what point can you do it or are you going to face a bottleneck because you still have a central whatever but that's a great example of where sometimes mitigations work and sometimes they don't So an example of something like your product is successful, if you're a startup or whatever, and you suddenly got successful, that success comes like a wave, right?
1:19:18Sam Newman: The wave of success hits you, and then a lot of it recedes. You'll get a thousand customers in, and maybe you'll keep a hundred. Maybe, right? That's just how things work. So you need to be up during the peak, but then you probably can shrink down again to keep the cost low. So if you've built your application to have some dynamic scalability in it, and if you're on the public cloud or a similar place where you can spin up and spin down, that's a viable mechanism, right? Okay, we can scale up to handle this and that's fine. Would I use that as defense against a distributed denial of service attack?
1:19:53Sam Newman: No, because then your distributed denial of service attack becomes a distributed denial of cash attack, right? You go from to not only having no service but also now no money because you hit spinning up, you know, nodes on your Kubernetes cluster and Amazon just keep taking the money. It's great. and you still can't keep up. So that's a perfectly valid mechanism for some things, but you kind of have to know the context in which to say, you can't just get yourself out of trouble. I think we're going back to a lot of times as an engineer, you want to know about the business. You want to know what creates a value, what creates revenue, what is important for the company versus what is not, and then where we want to maybe invest to with our infrastructure, with our reliability, or is this where it doesn't matter and it's not worth doing?
1:20:44Sam Newman: It's all about trade-offs. And I think it's about, you don't know, unless you get information out of your production systems, you don't know how close you are to the red lines. So that's why the observability is key, but it is all about trade-offs. And that's why I do a whole chapter on like formal methods. I talk about cap and pack elk and I say, look, these things are complicated, but actually really all comes down to a conversation with understanding the business context. So the example I give there is like if an e-commerce company if I want to buy an item online, right, physical item online, and if at that moment I want to purchase it, if you're not sure if you've got it or not, most e-commerce companies will say, yeah, sure, you can have it.
1:21:20Sam Newman: Because they know there's a good chance they do actually have it in stock. And if they don't have it in stock, they can get it in and back order it. And the worst case scenario, they can email and say, sorry, and refund you the money, right? So that's not great. It's not ideal. Now, if they said, we don't know if we've got this item, therefore we're not going to sell it to you, they're missing out on a sale. and I could go buy that somewhere else probably and they might have had it. So e-commerce companies in that situation, because of that sort of understanding of the relationship with the customer and the impacts if they get it wrong, they're going to try selling the item.
1:21:50Sam Newman: Keep that problem exactly the same, but now change the business context. Now we're selling concert tickets. So you're getting tickets to go see Tay-Tay, right? You're going to go see Taylor Swift and we're not sure if you've got the Taylor Swift ticket or not because our link to that system is down. But the Taylor Swift tickets are yours. you're taking your kids, taking your wife. It's all going to be awesome. We're going to Portugal. We're going to see Taylor Swift. It's going to be amazing. And then we email you. Sorry. We don't actually have those tickets. Now, the problem with that is, firstly, the devastation you're going to feel is a bit different.
1:22:21Sam Newman: They can't back order a ticket. And so in that situation, it's like the impact of the customer is so much greater. And then they'll book the flight. Well, you might book the flight. So what they do is make a different decision. So in that situation, if they don't know if they got the ticket, they don't sell the ticket. I'm not sure if you cover it in the book, but this is the fail open versus fail close, right? When you have an error or you're unsure, are you going to go through or not? And as you say, this can be the decision, the same problem. Two companies can take a decision. A company years down the road could take a decision.
1:22:50Uber had this classic example of when payment, we weren't sure if the payment went through. Do we allow the ride to continue or not? And early in the company's years, we always failed open. Yeah, just have the ride. We might have the money. and if not, we'll deal with it. As the company starts to go for profitability, we feel closed. We're not sure we have the money. If the charge went through, you're not going to get the ride. And again, same company, different focus. And if you're a developer, you needed to pay attention to what is important right now for the business. It was growth initially and it's okay if we lose money and then now it's, I don't know, unit economics, profitability.
1:23:27Sam Newman: But also that open close versus open, that's a failed close versus a failed open discussion isn't one size fits all for the whole system. it's on a feature by feature basis and even then you could say this taxi ride is 10 euros or it's a thousand euros you can make different decisions now if you approach this from a mathematical and from a computer science viewpoint we just talked about CAP theory that was a conversation about CAP theory right so in the chat I talk about here's CAP, here's Pac-Elk which is an extension of CAP I explain what those models are because when you get deeper into the business you'll hear about them but fundamentally that's what this is about Right, when you have a partition, that's what P in CAP really defines.
1:24:08Sam Newman: I can't talk to one part of my system. Remember, it's rule number two. When I can't talk to another part of my system, what do I do? And all you need to know as a developer, I can't talk to that. What do I do? Sometimes we can make technical decisions. I'll try again. I hope I don't bring my systems down for the retry storm, but I'll try again. If that doesn't work, what do I do? Well, actually, often the answer is, yeah, sure, you can look at CAP and PAC-ELC or whatever else. but go have a conversation with a product owner. What should we do? The resolution of that often ends up being a decision about, are we trading off consistency versus availability in pack alcohol harvest and yield if you're going into that version of brewer's conjecture?
1:24:47Sam Newman: But don't worry about that. That's in the book. That's why it's a later chapter. That's chapter like 12, I think, or 11. Don't start there. Start with having the conversation. Rule number two, I can't talk to this part of the system right now. What should we do? And that starts with having the chat. that starts having a chat. When we're talking about resilience, you talk about the dimensions to think about resilience. What are these dimensions? I start in the first chapter by coming back to that paper by David Woods I mentioned that was impenetrable to me at the start. And he comes up with four concepts for resilience engineering.
1:25:22Sam Newman: And I like the four concepts he comes up with because he helps us think broadly, right? So it starts off with robustness. This is where we can basically, I think in his words, absorb known perturbations. So we model a system and something goes wrong and we can kind of just deal with it in a way that none cares about, doesn't even know. So a very simple example would be in Kubernetes, a pod fails and Kubernetes means a replacement. Now the pod failing, that's a perturbation. It's gone a bit wrong, but we dealt with it. Great. So that's the start. And the thing is most engineers I speak to, they focus entirely on this.
1:25:56Sam Newman: They don't look at the other three. They focus entirely on this. One of the really nasty things about robustness is the things we often do to make our systems more robust in turn increases the complexity of our system, which in turn increases the surface area for new types of failure. I bring Kubernetes in to deal with the failure of a container. Now I've got Kubernetes and that becomes, and that's not solvable by the way. That's just something to be aware of. That is also why sometimes people don't go multi-region or multi-cloud because it's increasing the surface area of other failure types.
1:26:26Sam Newman: So that's number one, it's robustness. Can't always do one of those things. And for things to be robust, We have to know about them ahead of time. This is predictable. We deal with it. Then there's rebound, which is when something does happen and goes wrong. How quickly can we recover? So this is typically the time to recovery, well, time to detection and time to recovery. Yeah, and there's different ways of measuring that. And I talk about in the DR chapter, I talk about, you know, data loss. Disaster recovery. Exactly, yeah. But that's like how quickly can we get back up and running? So we've gone into a degraded functionality.
1:26:56Sam Newman: How quickly can we get up and running? So that's thinking about the rebound. and again for that you have to be very humble as a developer because the robustness is the system will never fail and it's like but what if we got it wrong okay well we have to admit things go outside and I guess for recovery on how you can recover this is where I mean if you want to do it well teams that are serious about this they do simulations of the system going down this is where Netflix is a very famous chaos monkey comes where it goes and shuts off parts of the system to like test this live and there's drills that are being done on how our on-call process works, how our decision-making, etc.
1:27:30This is actually a hard problem. And then you did it once, that's not enough because you need to do it regularly because the system keeps changing.
1:27:37Sam Newman: I'd like to give a shout out to Uptime Labs who are basically partly here in London. They do excellent instant training that you can do online in a web browser. It's fantastic. We'll add them to the show notes below. They're nice people. So you've got that recovery process, like how do we get up and running? And I think techies are with me, and then we go into the next two, which the third one is graceful extensibility. And so graceful extensibility, again, it's a slightly weird name, but it's basically dealing with surprise. When we make our system robust, we're dealing with the known problems.
1:28:09Sam Newman: Yeah, we know what to expect. What about things we don't expect? If we don't expect it, this is now not about computers, it's about humans. This is why distributed systems are fundamentally associated with technical systems, people and technology are working together. We couldn't expect the rabbits. How do we deal with the rabbits? How do we deal with the farm and the data center? It turns out badly is the answer. But if you're in a corporate environment, which is very command and control, quite hierarchical, where you've got a very narrow job description, you're often constrained in what you're able to do.
1:28:38Sam Newman: Those organizations tend to deal with surprise badly because the corporate structure is structured around the known. And so often structurally, you'll find that difficult. You mentioned drills earlier. Drills are a great way of helping with surprise. Coming in and having people often from outside saying, this has happened, how will you deal with it? Google had a thing, I don't know if they still do it, SREs would pop around and they'd do a thing called the wheel of misfortune. They had a physical wheel. Today, your system has suffered a spin the wheel. DNS poisoning attacked by a state, insert name of rogue state here.
1:29:09Sam Newman: Here's what's happening. How do you deal with it? That is all about people and experiences and trying new things and constantly pushing yourselves. I guess learning also from others. I remember this like very famous outage happened a few years ago at Meta where their DNS system through BGP, it just all went down. WhatsApp, Meta, everything. To fix this because they pushed out an update to their BGP table, which is inside DNS to their servers. Now to fix this, they needed to get access to the computer that physically to get access for whatever, because they couldn't reach it remotely. So they need to go into the building.
1:29:47Now, in going to the building, they had to go to a door that protected this system, which was running on their own domain, which was down. So they had to break physically into. But again, this is a thing where they share the details, which was great because now everyone else could think about like, do we have any of our physical security, for example, tied to our own domain? Because if you have it, you want to change that learning from what Meta have learned about.
1:30:13Sam Newman: And this learn takes us from graceful sensibility, which is dealing with surprise. That's like sort of almost in the moment. Are we ready to deal with surprise? Into the fourth one, which is sustained adaptability, which is how can we learn from what's happened? Not just the learning. So do we have the environment in which we can learn and share and grow, but then also adapt the system based on that learning? And it's called sustained adaptability because that is a completely continual process. That's not just today. That's not just when you have an incident. That's when you don't have an incident.
1:30:42Sam Newman: What are we doing to learn? And I think your point you raised there was excellent, which is I like reading other people's incident reports. There's a little bit of schadenfreude with companies I don't like, but there's also a little bit of it is like, they've learned something. Can I learn from them and do something that resolves my system? So when you don't have an incident, read other people's incident reports. There's a place called The Void. It hasn't been updated for a bit. There's like hundreds and hundreds of incident reports there which is worth going through. So that idea of sustained adaptability is all about us adapting and changing over longer timescales.
1:31:14Sam Newman: For those last two things, sustainability and greater extensibility, this all comes down to people and culture. Like having proper environment psychological safety is just absolutely required. If you can't raise issues, if you can't challenge in a safe way, you will deal with surprise badly because things won't get raised and you won't learn because things will be hidden. And so that all comes down to behavior stuff. And so the last three chapters of the book, we focus on that, is looking at some of the people and the social aspects of that. So I've got like 14, 13 chapters, or 12 chapters, sorry, about the other technical, and then the last three chapters are all around that side of things.
1:31:52Sam Newman: So like, what can we do to make sure we have that environment of psychological safety? Because without it, you can't actually be resilient. That's pretty amazing because so far throughout this whole conversation, we talked about building resilient systems. We talked a lot about people, about how to think about resiliency. One thing that we didn't really talk about just passively is AI, even though this book is coming out now. and I know it has a section on AI as well, but it's just interesting to reflect that even though AI is changing how we're building software, it seems like some of these ground truths of how to operate reliable software are a little bit just outside of this realm, right?
1:32:30Well, I mean... Or is it?
1:32:32Sam Newman: No, it's not. I mean, firstly, there's some AI stuff in there that can help. I detail quite a few examples around correlation analysis and help support around post-instant. there's some internet.io doing some amazing stuff in this area So you're saying we can use tools to do a lot, bits and bobs But really there's more fundamental kind of things here and the first thing is the macro and the micro at the micro level there's some useful stuff we can do here and I talk about in the book, the macro level is that none of it stacks up financially, right? So if you're taking a longer view about resilience, if I'm building my product based on a market sector that is fundamentally flawed these companies don't have a viable path to sustainability When you say which companies are we talking about?
1:33:16Sam Newman: OpenAI and Anthropik and the hypervisors, right? Conservative analysis by The Economist are showing that they need to be generating 2.6 trillion in revenue to justify their capex expenditure by 2030. The entire software market is 1.4 trillion. So macro level, I don't know how that gets sorted. I'm not smart enough. I also don't have any shares of those companies. But, you know, that's a separate conversation. I can't solve that. What I can start thinking about is, okay, if that's what I'm operating in, how do I deal with that? I can't deal with all that. What I can do if I'm taking a view to resiliency is being very careful about how I use AI.
1:33:51Sam Newman: For example, you know, if I'm thinking taking a longer view, being more flexible about model choices, that's going to help me because I've got the opportunity to use open weight models from different vendors. I mean, it's even just basic things like open AI and Anthropic are pretty terrible when it comes to availability of their core models from their services, which is why everyone I know also will use like, they'll use Google or Amazon or Microsoft as either their primary place they go to those models or their fallbacks. They have better uptimes and things. So there's all that stuff we've got to deal with at that level.
1:34:22Sam Newman: There's also the conversation about non-determinism and Charity pulled me up on this a bit. And I said, the thing is that, you know, we're going from deterministic software to non-determinism. you know that's a challenge for us we don't know if things work and then she said to me our distributed systems were never deterministic and now they're really not deterministic and of course she's right because when we take our distributed system and we add the people in beyond the rabbits right and the fires and the people we always had the degree of non-determinism we're just amping that up a bit yeah so i think that's a continuation so for me i think it's a confusing landscape it's not good or bad i think at the micro level there's some really great things that we can do.
1:34:59When we say micro level, that's the tactical things that we can do, right?
1:35:02Sam Newman: I think really what I mean by that is I'm taking the macroeconomic policy situation out of it. I'm saying as developers, there's things that we can make use of. There's some good stuff in there that we can make use of. But I also think there's some hedging we need to do. Longer term, if you're building AI into your product, there's some simple hedging things. So being multi-vendor, being multi-model, for example, breaking rather than having a single agent, breaking that down into more workflow models, Therefore, you can take bits of your workflow and realize actually this can now be swapped out for a bit of deterministic code that I write that could run more efficiently.
1:35:35Sam Newman: I think these are all sensible things that you can do. We can't fix this. We can at least deal with our world. And I think there's some sensible things that we can do in that space. What are the things that we know that we do not know about AI? I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is. Let's be real. Why did the LLM delete my database? Well, because it has no concept of causality. That's what it comes down to. It has no concept of causality. There's a glass here. If I push that glass onto the floor, you'll be working out as carpet down there.
1:36:15Sam Newman: If I ask you what's going to happen, you'll say the glass is breaking me fine. If I push it onto that floor, you're looking at it saying it's a hard floor that might break. You're carrying out a bunch of calculations in your head. You've got an idea about the physical properties of the universe. You've got a sense if I push the glass, this is going to happen. LLMs don't do any of that. They have no concept that if I do A, B happens. Well, there's a push for world models, but these LLMs are not world models. They're not world models. And world models have been around for a long time. They arguably predate LLMs.
1:36:43Sam Newman: They can do stuff like that. So they could make decisions like if you had the appropriate model and the appropriate inputs, they can at least get to the point of if I do X, then Y happens and I've been told not to do Y. this is why all the guardrails around LLMs are really not long-term going to be the right solution. And so I think we expect LLMs to do more than they can actually do because they seem so smart. And so I think that's the problem and that leads us to overestimate their capabilities a bit too much, I think. Or maybe over-rely on them without knowing and we might have some incidents later.
1:37:17Sam Newman: Absolutely, absolutely. And I think we're knowing where and when to kind of to use those things. And I think what we don't know right now is where should I be having, and again, this is wider, where should I have the LLM do it? And where should I just write some deterministic code? There's a lot to be said for just having your agent call a script and you know that script works, rather than having it implement the script in its magical brain for you. Well, there's something to be said that it seems both labs, both leading labs, Antropic and OpenA, have realized that coding is so important. So oftentimes when you ask the alum to do something, it will start to generate code because it writes a deterministic program, because that part will be deterministic.
1:37:57And it's very interesting when you ask it, it will often just pump out a Python or TypeScript or whatever, especially when it has to do with involving compare this or that. And they have trained these models to default to coding. I'm not sure how this came, but it seems to be working to some extent.
1:38:16Sam Newman: It's also because it's very tractable. right you know they are there to generate plausible code so they're there to generate plausible words right and in the context of code they've generated some plausible code that we can now validate does it execute yeah and so that's a really nice tractable aspect of it and there are some places where it works really really well but there are other areas where it's terrible like i when i write my book i use very little ai at all well you didn't use it for the writing use it for research. I don't use it for writing at all. I use it for research. But in the context of the research, I use notebook LM, basically.
1:38:51Sam Newman: But in that context, it's plausible. But then I just go and click all the links it gives me. And I say, was that there or not? I am checking the work. It's done way more work than I could have done going through all those papers. And God knows, there's only so many NIST papers you can read through, right? But in that context, I feel completely fine and able to do that. In other settings, you're getting now this problem called cognitive surrender, which is where you're using AI to write documents that you're never going to read. That's not really around a problem with the LLM technology. That's the problem with us.
1:39:23Can we talk about this cognitive surrender and cognitive depth?
1:39:26Sam Newman: Yeah, cognitive depth is a kind of term that's been around for a while. And that sort of explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools. And there's been a bunch of research into this. that's shown there's been some studies that showing the people's ability to think critically is impacted the more they use chatbots and things and i think we've sort of seen other examples that margaret and story's got a great um article about this and she cited some some of the research around it as well and that sort of explaining that i mean her her thing is you know she's talking we talk about technical debt building up in a system we have to be careful that our use of ai is that cognitive debt we need to have a shared mental model of what this system is supposed to do thinking about it narrowly for the lens of a program that we're building.
1:40:11Sam Newman: And if our use of AI, as an example, we all go off in different directions, that shared review breaks down. That becomes a debt cognitively. Cognitive surrender is a bit of a different thing, which is like, oh yeah, it's fine. It's basically LGTM, right? But for whatever the AI tells me. Yeah, LGTM was typical. Looks good to me. Approving large pull requests where users couldn't be bothered or you tried to look at it and for whatever reason you didn't want to go through or it came from someone who you trusted. I worked in the Google tool team on all that stuff, right? And so LGTM was literally a something with pattern match on and takes that as an approval.
1:40:50Sam Newman: So if you saw LGTM, that person's approved, right? And it was just, that was where I think that may even have been where the acronym originally came from. I'm not sure. I think it came from Google, yeah. Yeah, but it is this idea and it's become shorthand for I didn't really look at it. Yeah. And I think that's that cognitive surrender thing is coming out lots of ways. You know, we're getting AI to write documents and no one's going to read. And we've got AI that's reading documents that weren't written by other people. And so the cognitive surrender piece is really that the humans, it comes back to the human being in the loop.
1:41:21Sam Newman: There's very subtle differences in those workflows and to the kinds of differences it changes. Like there's a big difference, like Choredoctro talks about, it's a big difference, like if you're, say, a cancer, an oncologist versus cancer specialist. There's a big difference to you as an oncologist working and an LLM doing sort of detection of some of the scans that you're looking at going, oh, you should look over there, by the way, you might have missed that. That's actually quite useful, right? Versus the LLM-powered AI going, here's all the scans I've looked through, here's the ones I need you to sign off.
1:41:52Sam Newman: Right, the last one, the second one, right, takes the expert out of the loop and makes them an approver. And often in that context, they've got rid of a bunch of their oncologists. You've got one person in the funnel that's going to lead into issues, right? The other model is we're making people better. I think that's to come back to the original idea of AI or some of the pitch, right, was it was going to free us from drudgery. Just like the Industrial Revolution was going to free us from drudgery, but like it would elevate us. We'd have less toil. We can focus more on critical thinking and smarts.
1:42:22Sam Newman: And so much of our current use of AI is not freeing us up from critical thinking. It's causing more context switching. It's causing less time for thinking. People are working longer hours. We've got studies that show this now. That's not necessarily AI's fault. It's just how we're using it. It's like, we need to readdress that a little bit. Can we talk about complexity and AI maybe adding it? I'm going to quote you from a few months back. You wrote, I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space.
1:42:56Sam Newman: Yeah, I can't remember the exact context in which we had that conversation. But I mean, the platform space was all, I mean, the platform space is kind of interesting, right? Because that was an attempt to make things simpler for developers because Kubernetes became popular and it's not developer-friendly. That's not what it's designed for, right? But most developers never have the touch of it. No. I mean, well, if you have a platform. If you don't have a platform, yeah. Yeah, there's a platform team's problem, right? And that's why they get paid. And so I think there was this idea that I was sort of hoping that there'll be a bit of a complexity collapse around that space.
1:43:27Sam Newman: I had this hope for things like WSGI, for example, simplifying down those programming models a little bit. And I think there's starting to be a bit of a conversation starting to be had about some of that stuff going on. What then happens, of course, is AI has come out and everyone's like, I've got to AI all the things. And so rather than thinking about what we need and trying to strip that complexity back, we've now just got another piece that requires us to still have all the old complexity of adding new complexity on top. And somebody's got to look after all that stuff. Yeah, but I think it goes back to a little bit of like, we seem to have more work with AI.
1:44:01And probably it doesn't help that it can add more complexity, especially when we're not being clear of what we want, we're not being critical of the output, when we have cognitive depth or cognitive surrender, especially not with cognitive surrender, you know, just right. So these things might be amplifying the fact that it's just more work, more complexity, right?
1:44:19Sam Newman: Yeah. And I think there's also that connection through to what the AI is doing. It still comes full circle a little bit, which is the idea of potentially not looking at the code isn't to say that you don't care about what the system is that you're building. It's to say that you've got a better way of understanding what you're building. So one of the concerns is if I'm using AI to create code, this is going to create awful loads of debt and a complicated system that no one's going to understand. Well, we can't let that happen. We still need to remain connected to the system that we're building, which is why I'm a big fan of modularizing systems.
1:44:51Sam Newman: Whether or not you put those modules in different computers, I don't actually care. Getting AI to work within the modules, having the humans think carefully about what those modules are and the connections between them let the ai go hog wild inside those boundaries that are easy to change the gaps between them that's where the human beings spends to think about the mental model of how things fits and slots together um that feels a very sane way for us to balance our time and our energies appropriately so do i understand correctly that you're saying that it's probably a sensible approach is have the architecture in your head, know how your system works, know how the boxes work, know where there's AI inside it.
1:45:29Maybe it's generating code. Maybe it's running your incidence response in isolated ways. And then spend your time talking with other people in the business, on your team, with your customers because you're, as you said, systems like software is people's ideas connected and you have things in your head or you write it down. But now using AI as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions because maybe you have more time to, as an engineer, understand more on the business, on what is working, what is important, those kind of things.
1:46:02Sam Newman: Yeah, and also to make the structure on the model system a bit more explicit as well. I mean, that's one thing a good modular structure does. I mean, this is why Shopify have been so successful keeping their thing as a monolith, right? It's because they spend a lot of time building their monolithic application as a modular domain-driven architecture, right? because it makes the structure easier to understand. You can zoom out and see how those things fit together. But also in the short term, while we're having these debates about are we going full software factory or not, we don't have to make a one-size-fits-all approach.
1:46:35Sam Newman: We can say, look, this is our architecture, being a modular monolith or a microservice architecture, it doesn't really matter. And over here, this area here is where we're trying a full software factory approach, where we don't look at the code anymore, but we specify the interface. We have verification around that. We know what good looks like for that module, right? We've got functional tests, tests that look at, say, some of the operational requirements as well. These areas over here, we're still grappling. These are really key and really important to us. We're not ready to give up on those yet because we don't have the trust level yet.
1:47:05Sam Newman: We don't want to take that risk just yet. And so over here, we're going to work with the AI more of a collaborative way while we work together. And we're still going to be intimately involved with the code because we're not ready to make that step here. But if we approach our whole system with one big black box, then we're forced into making one or either decision. I don't think that's healthy. Do I understand that you're saying that you think that the software factory, the dark factory, you're giving it a spec or the AI either building it or operating it, that you see that as viable? There are some companies I know that are already all in on this.
1:47:36Sam Newman: Not everyone should be doing it whatsoever. I think if that is our future, if that is our future, I think it's a lot of ifs there. The question is about how you go about adopting and trying. So what I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way. And like low-risk environments and maybe simpler domains, those kind of things, right? But also like a lot of dark factories don't create finished products. Like real world, beyond the fact that real world dark factories, there aren't many of them, most of them create components. They don't create the finished product.
1:48:07Sam Newman: So in the same way, if you think about a module being a piece of the overarching system, right well let's have a factory around these bits here and actually looking at it like if we had to rewrite that from scratch it might take us like with the collaborative way with ai it might take us a few weeks to have to rewrite that if we had to so actually we're quite relaxed about leaving that to go as is and it's been working fine and it's a it's a module around which you've got great test coverage already for example where it's not on the critical path from a latency point of view or it's a service where we've got really good observability data and really well-defined slo's and we've got a good steady state, well, we can start running it as a factory now and seeing what's happening in production.
1:48:47Sam Newman: We can have a look at the SLOs. We maybe add some special SLOs around there. I'm hearing correctly that you're saying that as a professional, as someone who wants to be up to date with the new techniques that we can use, it's probably, if you have the opportunity to start to build a software factory, a somewhat autonomous place in a component, in something low risk, it's probably worth trying it out to see how it works, to see what you learn, to see if it even works or where it breaks down. But do it in a way where you're minimizing the last rate. It's a little bit like when, you know, when microservices came out, the concept, you didn't start by moving your biggest thing.
1:49:22You started to build a small service that didn't really matter all that much. And you were okay to learn from it when it goes down and it wasn't on the half path for the business.
1:49:31Sam Newman: Yeah, I think it's the thing to start off with is getting your head around spec. So looking into spectrum development, whether that's Kiro or TESOL or SpecKit, whatever that is, I mean, they're all slightly different. Have a look at Brigitte's article on that. It's a good on-ramp to get ahead of it. Start off by playing around with that because that's the stepping stone. That's like, you have to define what good looks like as an input to your factory. Yeah. Start with the spec. Good look input and also for the output. Yes. And then it's like, how are you verify? How are you going to go about verifying the output from the factory?
1:49:58Sam Newman: Have a think about that. And, you know, like, not just in terms of the kind of normal sort of behavioral aspects of it. But I agree with Kevin Henley on this. He said he hates non-functional requirements because if a functional, if something is non-functional, it means it doesn't work. So he breaks the world down into like development requirements, operational requirements and sort of business requirements. So I think it's a nice way of doing that. So along those three axes, how do I verify that the software being created by this module has matched my developer requirements? So for say maintainability, my operational requirements, latency, uptime, and my business requirement.
1:50:34Sam Newman: Is it working? Features working. So as long as you've got a way of validating do your software around that and if you don't know what the good looks like for those three things guess what this isn't for you but you can still do specs by the way but you can't do a factory yeah but you can do specs and that's your edging and your way there and i think it might not work there may be some there may well be some blind alleys this could be model driven development all over again which we do not have time to get into because i get very angry about it but it could be that but i have a hope it might actually work this time unlike mdm and then as closing we i think a lot of what we talked about is it's important to hold the architecture in your head to think about the architecture thing how to modularize now for engineers who want to get better at this kind of high level thinking strategic thinking architectural thinking outside of reading your book which i think brings a lot of really good ideas on architecture as well and not just resilient systems.
1:51:31But outside of that, what would your recommendation be for a tech professional, for an engineer who's like, I'd like to get better at architecting systems to know what good looks like?
1:51:41Sam Newman: I think there's a few books I can point people at. I think the Software Architecture Fundamentals by Neil Ford and Mark Richards is a good read. Just to be clear, they're friends of mine, so I'm not unbiased in recommending it. I would say around the modularity, Vlad Kononov's book on balanced coupling is fantastic. and his book on DDD is also very good. But that balance coupling book is really great. He's also developed some skills in Claw to take some of those ideas. So definitely, definitely take a look at those. I think if you're looking at things architecture and you think in terms of modules, for me, I don't agree with everything in Mark and Neil's book.
1:52:14Sam Newman: There's lots of great stuff in there. But for me, it actually all starts. I sort of mentioned earlier that writing the second mission of building microservices was really all about smuggling in, getting people thinking about single information hiding. It's a very old idea for the 1970s. It still comes back there to me. modules are still the whole thing for me so i would read vlad's book and if you want to go read the original parnas papers from 1971 1972 they're still worth reading but those would be good jumping off points i would say you can still i think the nice thing about thinking in terms of modules is even within an existing code base you can start to modularize it in some small ways you can start to think about boundaries you can think about clear separation about using encapsulation as a way of information and i guess it's fair to say like look if you have an idea on how you can make your existing code is a bit more modular, you have zero excuse not to do it.
1:53:02I mean, we don't trust agents fully, but this is a textbook example of like if you can say, here's my goal, refactor this, like if it would have taken by hand X amount of time, it will be faster. So you should be empowered to at least experiment with these, right?
1:53:17Sam Newman: Especially if you've got a decent test suite. And if not, start with that. If you've got a statically type language, you don't even need to, just a good old-fashioned IDE does an amazing job, right? Dynamically tight language now don't have the excuse, especially if it's a modern one that's well understood by the NLM. So if you're using, you know, Python or Ruby, it's so much better now than it used to be for refraction as code bases. Absolutely. You need to get a test suite in around that. But yeah, you can start modularizing. And that's about building. And that's also a little bit coming back again to taking, we don't want the architecture only in your head, right?
1:53:48Sam Newman: We want the code to sort of represent that mental model. Yep. Because we communicate through the code. maybe in the future be communicating through the specs and when you make those things more explicit then i think that helps everyone stay on the same page there's uh george fairbanks wrote a book i've forgotten the name of the book now but in it he has a pattern called an architecturally evident coding style and that we should code in such a way that it clearly communicates the architecture of the system and i think our programming languages are quite bad at having a good concept of modules in general.
1:54:23Sam Newman: So I think we have to go a bit above and beyond ourselves. But we could do more to make the architecture of our systems that little bit more explicit. This is also why I spend time production, because especially if your modular structure ends up being a process structure, in other words, it's process boundaries. I can go look at the networks to find out my architecture. I shouldn't have to, but at least, you know, I know production is truth. It's like football is life, production is truth. So where can people get your book? You can go and order it now. if you can go and buy it from a local bookshop that's always a great idea it's also going to be available for people on the O 'Reilly platform in sort of digital form so you can go and read it as part if you're a subscriber at O 'Reilly you can get it there as well but lots of people love having the physical copy and we have these in the show notes below so check them out to get the book directly and also you've been doing some videos right?
1:55:12Sam Newman: Yeah well I've been doing some stuff on Dave Farley's Modern Software Engineering channel it's kind of a good excuse to do some sort of semi-ranty you know monologues to video and have some fun chats. So yeah, go for the Modern Software Engineering channel on YouTube and you can find videos from me and Dave Farley and Kevin Henney's there. We've got stuff from Kent, Trisha G, Emily Borch, Dan North is there as well. I'm pretty sure I've forgotten somebody else. There's almost a couple of other people there as well, but we try and put videos out once a week and it's a bit of a fun place. And we're also looking at that one.
1:55:44Yeah. And if people would want to work with you, where can they find you? How can they find you? What kind of stuff can they get help from you for?
1:55:52Sam Newman: I do like internal talks with people, like internal events. I do like customized training courses and a whole variety of things around resilient systems and security and microservices. I also do a lot of consulting for clients, helping them around sort of re-architecting their systems and stuff like that. If you know more about that, actually go to my website, samnewman.io and you can contact me there. You can also find me on LinkedIn. But if you come straight to the website, use email. I'm still a bit old-fashioned like that and I work with clients all over the world. So if I'm able to help, yeah, head over to samnewman.io my contact details are there.
1:56:22Wonderful. Well, Sam, thanks again. This was great.
1:56:25Sam Newman: Thank you. Thanks very much to Sam for this conversation. I really like Sam's three rules of distributed systems. One, it takes time. Two, sometimes the thing you want to talk to isn't there. Three, resource pools are not infinite. Sam said how most of the outages happen because of this last one, resource pools not being infinite and services are getting saturated. But I was chuckling when when he was talking about rabbits stirring through networking cables between office buildings. Then again, a few weeks ago, with thousands of flights stranded on the East Coast of the US, it was because workers accidentally cut a telecom cable in New Jersey used by the FAA.
1:57:00Another food for thought from Sam is how failure should be handled differently, depending on the type of business. An e-commerce shop might decide to show you the item when the inventory service is down. Worst case, after you order and pay, they just refund you because we're like, oh, we don't have the item. Basically, failing open is fine in this case. But on a concert ticketing website, you don't want to sell a ticket that might or might not be there. And so failing clothes is the right call to make. Because again, as we discussed in the episode, if you sell the ticket, people might book flights, they might get upset, you might lose them as customers.
1:57:31Finally, I appreciated how level-headed, perhaps even cautious Sam is about AI. He's got no doubts that AI is very useful at the micro level, basically a tactical, a things that we can already improve. But he is worried about the economics of the model providers. His suggestion is pretty pragmatic. Hedge your vendors and your models. Be multi-vendor and multi-tool and wherever you can, swap non-deterministic AI agent workflows to deterministic ones. Basically, have LLMs generate deterministic code, which I find very, very refreshing thinking. Check the show notes below for the story about scaling Uber in the early days with Uber's first CTO, Tuan Palm, that we discussed, and it's an awesome story.
1:58:09And for more, the pragmatic engineering deep ties on resilient systems. If you enjoyed this show, do subscribe to your favorite podcast player. And thank you very much if you recommend the podcast to your friends or your colleagues. Thanks, and I'll see you in the next one.
Brought to You By:
• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.
• Entire – Git hosting, rebuilt for the agentic era. Every agent session, prompt and tool calls: stored in your repo.
• Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages.
—
Sam Newman wrote one of the most-read books about microservices (“Building Microservices”), but he calls them an architecture of “last resort.”
In this episode of the Pragmatic Engineer podcast, Sam explains his thinking on this, and he’s certainly well placed to do so; he was in the room when the term “microservices” was coined. We discuss what teams get wrong when adopting microservices, why independent deployment matters, and how microservices can help teams work more autonomously.
We also delve into his new book, ‘Building Resilient Distributed Systems’, and Sam explains his three rules for distributed systems, why observability is essential, and also why it’s vital to take business context into account when deciding whether to fail open or fail closed on errors.
We also explore how AI is changing software development, from specs versus code as a source of truth to cognitive debt and cognitive surrender. Sam shares how modular architecture can help teams experiment with AI while maintaining understanding of the systems being built.
Timestamps
00:00 Intro
03:16 Sam’s path into tech
09:32 Thoughtworks
20:55 The rise of microservices
33:37 Are specs becoming more important than code?
45:22 Building Resilient Distributed Systems (Sam’s new book)
52:35 Three rules of distributed systems
56:16 Observability
1:02:20 Resilience tradeoffs
1:07:54 Idempotency
1:15:58 Thundering herds
1:21:03 Business context and resilience decisions
1:25:51 Resilience engineering: four concepts
1:32:42 AI and resilience
1:36:26 AI’s limitations and where to use it
1:40:06 Cognitive debt and cognitive surrender
1:45:02 Modular architecture and AI software factories
1:51:53 Resources for learning software architecture
1:55:30 Where to find Sam
—
The Pragmatic Engineer deepdives relevant for this episode:
• Scaling Uber with Thuan Pham (Uber’s first CTO)
• What is good software architecture at Netflix?
• The past and future of modern backend practices
• What is reliability engineering, and the history of SRE
• How to debug large, distributed systems: Antithesis
• Designing Data-intensive Applications with Martin Kleppmann
• Building Bluesky: a Distributed Social Network
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].