1. Podcasts
  2. The Stack Overflow Podcast
  3. It’s ok that computers are (sometimes) better than us
The Stack Overflow Podcast

30 Oct 2026 · 27m · 10 chapters

Episode artwork

It’s ok that computers are (sometimes) better than us

In short

Applying “immutable infrastructure” and cattle-style disposability to software, arguing that software cost is driven by maintenance and understanding, and that AI enables scalable evaluation (evals) of many code variants. Also covers governance, “boring software,” resilience vs deterministic specs, and the line between observability and surveillance.

Guest

Charity Majors, co-founder and CTO at Honeycomb (observability). Background: grew up in rural Idaho; studied piano; shifted into computers when Ethernet wiring began in dorms; early sysadmin/on-call experience; credits Chad Fowler’s immutable infrastructure work (coined term in 2013).

Key claims

“Mutability is the sworn enemy of understanding”; re-architect around bounded functions and durable evals; replace “strangler fig” with more efficient eval-driven regeneration; AI is logic/language but not higher-level systems thinking; governance must catch up; observability should target machines, surveillance humans.

Notable examples

YAML-defined infrastructure; ChatGPT skepticism then acceptance; strangler fig rewrites; latency tolerance (2s→3s ok vs router hop 0.1s→3s catastrophic); NSA/Facebook data-gathering concerns; HIPAA as a governance model.

Written by AI. May contain mistakes.

Chapters

0:45

Charity Majors' Journey into Tech

Charity shares her unconventional path to software and technology.

“And I went to college and pretty quickly realized that all the music majors who had graduated were still poor in their 30s and 40s, still hanging around the music department.”
1:08

From Pets to Cattle in Infrastructure

Discussing the concept of treating code like cattle rather than pets.

“And you could argue just swapping one keyboard for another.”
2:32

Understanding Code Maintenance

Exploring how immutability in software can reduce complexity and maintenance costs.

“by around you know early 2000s it became very clear to everyone that this was better it was better because you know one of chad's great lines is uh mutability is the sworn enemy of understanding.”
4:28

Re-architecting for Efficiency

Discussing the architectural changes needed in software to enhance understanding and flexibility.

“because code has always been hard and time-consuming and expensive to produce.”
7:16

Governance and Boring Software

Addressing the governance issues surrounding AI and the importance of creating reliable software.

“But I talked to a QA engineer that had built a sort of proof of concept, agentic SDLC, and it was all based on evals and QA mindset.”
11:00

Becoming Indispensable in Tech

Strategies for professionals to become indispensable and thrive amidst AI advancements.

“And I think that's where we would start.”
14:02

The Role of AI in Personal Impact

Discussing how individuals can harness AI to become indispensable in their fields.

“I can have more of an impact than on anything else.”
19:10

Surveillance vs. Observability

Exploring the distinctions between observability, monitoring, and surveillance in technology.

“that I think you're the right person to ask.”
20:43

Engineering Mindset in Society

Analyzing the implications of applying an engineering mindset to social systems and legal frameworks.

“But, you know, engineers are smart people and they think that they can solve things that are outside the domain.”
22:30

Lessons from Meta's Fast-Tracking Era

Reflecting on the lessons learned from Meta's approach to innovation and leadership.

“But I think what's going to shift is that, you know, the SaaS that is indispensable, the stuff that's like, you know, a glorified database.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On The Stack Overflow Podcast, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:08Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today we are on the floor at the LDX3 conference. We're talking about cattle, not pets, but for code. And my guest for that today is Charity Majors, co-founder and CTO at Honeycomb. So welcome to the show, Charity. Thanks for having me. My pleasure. So on top of the show, we'd like to get a little familiar with our guests. Can you give us a little TLDR on how you got into software and technology? It was an accident. I went to school for piano when I was 15, 16. I grew up in rural Idaho, and the only thing there was to do was read books and play piano.

0:49And I went to college and pretty quickly realized that all the music majors who had graduated were still poor in their 30s and 40s, still hanging around the music department. And I was like, no. And that's about the time that they were wiring the dorms for Ethernet. And I started playing around with computers. And now I'm here. Well, you're still in orchestration, right? I'm still doing orchestration. And you could argue just swapping one keyboard for another. There you go. You gave a talk yesterday that was really interesting. And it was sort of an idea that I haven't heard. We're taking the idea from infrastructure, where we went from individual servers as pets to moving to disposable infrastructure as cattle.

1:28But you said we should do this for code. Yeah, and I have to give so much credit to Chad Fowler here. Chad is the one who actually coined the term immutable infrastructure back in 2013. And he's written a lot about this blog post on the Phoenix Architecture site, and he's writing a book for O 'Reilly on it right now. But it was one of those things that as soon as I saw him make the case, it clicked in my head, and I was like, oh, yeah. I am just barely old enough that my first job title was sysadmin. Right. I barely remember this stuff. But I do remember the sense of crisis that everyone was in.

2:03I was 17 years old and on call for the first time. I was surrounded by these people who were just wizards. They knew everything about Linux internals, FreeBSD, HPX, Sun, like the operating system and the networking stack. And they knew everything. And every server was like handcrafted. crafted like you would ssh in and be like which type of memory do i need to fix the hardware like they knew so much and it seemed unfathomable to me that we could just destroy those yeah and yet by around you know early 2000s it became very clear to everyone that this was better it was better because you know one of chad's great lines is uh mutability is the sworn enemy of understanding.

2:45Anytime you're going just changing, you lose the provenance. You lose the history of what's come before. And it's not repeatable. And the whole way to make technology boring in a good way is to make it so that you don't have to understand it all the way down to the device drivers so that you could work at a slightly higher level because you know how to revert to something that's known to be good. And so Chad started laying out this argument and it made so much sense to me. And a lot of this does hinge on what is the definition of the word spec. because it's a catchy phrase, but I think the definition of spec is changing a lot.

3:20And I just feel in my bones that there's a better way. The cost of software is defined by maintenance. Yes. So I want to tap on the immutability because I think for a lot of people would look at the server and that being like, that's the immutable infrastructure, right? Once you start defining infrastructure with a YAML file, you never know exactly how many nodes are up. That seems like that's kind of mutable. That's kind of like I know less about what's going on. Is that mistaken or is that just like you said, another layer of abstraction? I think there are parallels here that history isn't going to exactly repeat itself.

3:58That's just moving up the stack, you know, like going from assembly language to C to JavaScript. And I think it's notable that there have been many movements in the past who have low code or no code, and they've all kind of failed. Which is why I think it was very reasonable for us to be very skeptical of ChatGPT's claims right up until about last November. Right. And you were a skeptic for a while. I was a skeptic for a long time. And last year, it's here. And at its root, this is almost more of an economic argument than a technical argument. because code has always been hard and time-consuming and expensive to produce.

4:34And we've always been rate-limited by some human has to understand this before we can move forward and iterate on it. And if you think like I do, that in the past year, AI got to the point where it can generate snippets of code that are roughly at the same level of quality as the median software engineer, which means that while one software engineer can output a piece of work that works, AI can generate a thousand of them, ten thousand. We can see which one is the best. It makes you think very differently about code itself. It's interesting where you spin up all these nodes, now you can spin up all these versions of that feature and compete them against each other.

5:16That's one of the building blocks, I think, because it's pretty clear to me that in the same way, We don't expect every software engineer to understand how to debug types of RAM or how many CPUs or if there's a device. We just don't expect them to understand that. Right. You know, we used to. Right. We don't anymore. And I don't think it's an exact analogy, but I do think that when we talk about cost of software as defined by its maintenance, we're talking about the cost of having to understand everything. And I think that if we re-architect our software so that we can understand it at the level of functions instead of lines.

5:51Right. You know, if you re-architect it so that there are bounded parts that can be cut. And, of course, the most durable part of this system would be the evals, not the code itself. Right. The evals describe what must be true for this to be defined as good. And then, yeah, we can generate 10 ,000 diffs and see which one is the best, you know. Yeah, yeah. And for those evals, are you thinking like the LLM as a judge sort of evals, or do you have a deterministic eval? Well, both. And I mean, a lot of these tools actually already exist. Conformance testing, they're all in the realm of ops and QA, which are historically like the cost centers.

6:26You're not quite as cool as us software engineers. But like they're aimed at describing the world as it is instead of the world as it should be. And there are some types of code that will never be replaced by this because they're too sensitive. But there's a lot of code that's very high in the stack and it's a feature. You should be able to say, these are the contracts that our code has made. These are the systems that depend upon it. And right now, the fact that we don't know those things, we don't know. Like the way that we replace old code is through strangler fig architecture, where we just intentionally break one thing at a time.

6:55This is terrible. That's why it's so hard. Just regenerate the architecture. Jez Humble had a talk yesterday sort of specifically targeting the strangler fig architecture. Ah, I'm going to have to catch that. Yeah. It was interesting because it's like that's a really inefficient way to do it. Yeah. It's just screen tests all the way down. It's no fun. The worst engineering times of my life have all been rewrites. But I talked to a QA engineer that had built a sort of proof of concept, agentic SDLC, and it was all based on evals and QA mindset. I was like, oh, engineers are going to need to be a QA person.

7:28Yes. And ops. One of the things that I'm excited about in this era is software engineers have historically lived so much in the land of how things should be. If the tests pass, my job is done. You know, as much as we say the lob it over the wall doesn't exist, it still exists, right? We're used to living in this world of, well, it is correct on paper, but that's not reality. The code has never been the actual source of truth. The source of truth is the system. What matters is the system and its ability to keep on running in all situations. You mentioned that the nature of specs has to change. Talk to folks who are thinking about spec-driven development and what that spec actually is, whether it's a sort of functional spec languages or plain English.

8:12This is not really my area of expertise. And I think it's just a really exciting time to be thinking about those problems if you're someone who likes those problems. One of Chad's recent posts said something like, don't assume that spec is a document. Like, there are probably going to be bits of code in it. There are probably going to be, like, depends on the type of system you're trying to describe. There might be benchmarks. There might be a lot of different things. And the tools don't really exist to do this type of development today. Sure, yeah. But they're starting to be built. Yeah. And it is very exciting.

8:41I briefly ran into somebody and mentioned this sort of regenerative code. And it's like, well, it works for infrastructure because you have that deterministic thing, that YAML file that generates the same thing every time. Do we need the thing that generates the same thing every time? Oh. You're sort of talking about like this code thunderdome. We need our AI to generate deterministic system that meets the specifications. But I think that it can change. This is part of the challenge and the promises, right? What can change and what can't? If you've got a web app and the latency on the payments endpoint goes from two seconds to three seconds, probably fine.

9:19It should be fine. We don't know until we try it, but that kind of variance happens all the time for a lot of different reasons. And you need systems that are flexible enough to absorb some variance. Something that is much lower on the stack. Like if you had a router that went from taking 0.1 seconds to taking three seconds to take a hop, that'd be catastrophic. That'd be terrible. Right, right. Right? But like part of architecture and part of building resilient systems is trying to make them flexible so that things can flex. Right. You know? I'm sure a lot of devs listening to this will be like, oh, the spec is never a deterministic thing.

9:58The requirements shift over time. Exactly. Yeah. So, at what point do we need to get the final final, right? Like, how do we stop product managers from changing things along the line? You know, I think it will be great if we can get to a world where we trust our spec and our evals enough that anyone can just chip code. That is not the world we live in now. And it's not one or two hops away from the world we live in now. Right. But for some types of systems, you can see how it might be three or four hops. One of my favorite blog posts of all time was the one about boring software. Yeah. You know, boring doesn't mean bad.

10:40Boring software runs the world. It means that its failure conditions are well understood. It may fail, but we know what to do. In the same post, you talked about innovation tokens. You know how you need to limit the number of... Every startup gets two innovation tokens. You know, spend them wisely. And none of this stuff is boring right now. But a lot of the technology that we would use it for is boring. And I think that's where we would start. How do we make AI boring? Ooh, that is a topic that really starts to come down to governance really quickly. And I was thinking the other day, five, six years ago, there were plenty of large language model research companies out there that were doing research on their models and following the law when it came to copyright and fair use and everything.

11:25and they were making progress, not as fast. And then OpenAI came along, just like, just go. And how different would the world have been if the changes since 2022, 2023 to 26 had happened over 10 or 15 years? And what kind of precedent does it set for all of the presumably law-following companies out there? Sure. So something that I've been thinking about a lot lately, this is not necessarily technical, but just like how do we catch up from a governance perspective? How do we catch up? There's this upsurge of rage at data centers in the company throughout the country right now. But like how do we channel some of that into organizing to, I don't know, compensate the people, have funds for the people who want to move because they can't sleep at night.

12:11It feels like this is the question of our generation right now in so many ways. I mean, it does sort of feel like AI is just feeding society into a wood chipper to get this amazing sawdust, right? Yeah. Yeah. There's a Cal Newport just like, why are we letting these folks act like, oh, it might kill everything, but we're going to build it anyway. Right. Like, this is a product that's broken. Like, if your product was out there, like, putting people out of work, would you feel okay building it? Developers have a different view on AI because they can engineer solutions to their problems. Society can't.

12:46Well, and this is the thing. I am very much of two very strong minds on AI because as an engineer, I'm like, oh, my God, it's Christmas morning. Right. This is so cool. Right. And I really do think that software will turn out to be the killer app for large language models because it's logic and language. Yeah. It's what it is. The LLMs are getting so good at, you know, if you give it a task, it does the task. They're not getting any better at the higher level systems thinking. And I don't think they will because of the nature of the thing. Like it's still just sort of predictive text. It's trying so hard to be helpful.

13:19You know, we had an article from Chelsea Wolf that was like, data-driven thinking is not innovative thinking. And this is LLMs or data-driven thinking. Yeah. Right? A lot of the rhetoric about how we're all going to be out of jobs and everything, I think is, I wish people would calm down. I don't think it's going to be true. But on an individual level, if you don't embrace new technology, yeah, you will be out of a job. Yeah. So like both of these things are true. And back to your question about how do we make AI boring? There are individual levels, there are company levels, there are societal levels.

13:51And I think none of us can afford to lean out on all three. You know, sometimes when I lay awake at night thinking about climate change and the government and everything, I do find myself thinking, well, when it comes to AI, I can have more of an impact than on anything else. yeah and it's small but i'm in it and you're in it and we have voices and it's the game is not over so there are things we can do right and we should be super actively leaning into that opportunity for folks who want to lean into it what is the thing that the i see listening what can they do the first thing i know this not everyone thinks this is become an expert become someone worth listening to you know it's so much harder for people to like discount your voice right if they need you.

14:36Become indispensable. Master the tools. Develop your opinion. Become indispensable is a great way of summing it up because not only will you be needed and you'll be listened to more for social animals, you know, but also you're more likely to come up with interesting ideas. You'll know the weaknesses. You'll know where it shouldn't be used. You can make the case. It's credibility. Yeah. The other things I've heard that will make you indispensable are things like taste and there was a talk yesterday that said it's wisdom and charisma. Intelligence is a commodity. I've been thinking about that a lot.

15:12I just finished reading this book by Benjamin Labatt. I read it, I got to the end, and I turned around and started over from the beginning. I've been thinking about the Move 37, where it is this entirely unpredictable move. It looks innovative, but it's from the simulations and the data. Like, the computer can do all of the moves forever. For those who are listening, it's sort of about von Neumann's life. I was watching an interview with the author and he was asked about his process and he says it's very research heavy. He starts with research. He tries. It's shelved under fiction, but very little of it is fiction.

15:47Most of it is taken from the data, taken from history books. And it's sort of like a dark origin story of artificial intelligence. Yeah, yeah, yeah. It's interesting, you know, the personality. Like, von Neumann was very good at solving problems. He would just have these people come by and just, like, chat with him and be like, hmm, solve their problem. And he worked on the Manhattan Project. He worked on computers. He worked on the hydrogen bomb, the nuclear bomb. He built one of the very first computers. Yeah. Like, anything that happened, like, the first few decades of the 20th century, like, his figures were probably in it somewhere.

16:24What about that book struck you for this moment? So many things, but it's written as sort of a triptych. The last section is about AlphaGo, basically, and the Go players. And the guy, I don't remember his name, the Go champion who played, and he quit the game a couple years after losing and just being like, ah, there's no point. And I kept thinking about that, and I'm like, yet if your entire life is oriented around being the best and the smartest, This is very threatening. Right. But most of us are not the best and the smartest. Most people are all average in some way. And I feel like... Embrace your mediocrity.

17:04Intelligence is overweighted on. It has been dramatically overweighted on for the past few decades at least, if not longer. There are so many things about being a person are not about how smart you are. And the more obsessed we get with our intelligence and with being the best, the more we really lose out on the journey and the joy of the like there's so much more to life than smarts yeah we all have become knowledge workers right our power in the labor market is our knowledge and with ai i mean computers one of the other things i liked about this book was that it showed how how much it scared people the advent of quantum physics yeah when we cease to understand the world yeah and he was just like they really lost their like these mathematicians and physicists who were like, we can no longer model how the world works.

17:54And it's math instead of intuition. And some of them actually went insane. Yeah, it was kind of a mystical experience for a lot of them. Yeah. And now we're all just like, well, yeah, quantum theory exists. And it's fine. And there are so many other types of AI that are so much better than humans at so many things. Driving. It doesn't have to threaten us. I think we need to have the time to adjust. The speed of this is so destabilizing to everyone. And the speed of it and the fact that people seem to be gunning for terrible things with big grins, just like with glee. They're just like, we're going to put you all out of work.

18:25Isn't that great? It's just like, do you trust these people? No, I don't trust anyone who says that. It's a terrible thing. But I don't think it needs to or should be intrinsically terrifying or threatening to our sense of self. The computers are better than us anymore than it should threaten me that someone else is a better CTO than me. I can want to be like them. I can make a problem with stuff after them. But it doesn't threaten me. because I am who I am. I can still love my family and cuddle my cats. And there are so many things that are not intelligent based. And I wonder if that's kind of been the point all along.

19:00I mean, maybe. I don't know. It's an interesting take to say that, if I'm understanding, AI will sort of rehumanize us. I hope so. I hope so too. So I have a sort of spicy, provocative question that I think you're the right person to ask. Oh no. Yeah. Yeah. What is the line between observability and monitoring and surveillance? Off the cuff, I would say observability and monitoring are pointed at machines and surveillance is pointed at humans. Well, like you said in your talk yesterday, the software is a system and that system includes users. Which is why we have things like PII and types of data that we're supposed to scrub so that any old person off the street can't come in and just like see where I was at a given time of day.

19:42Right. But in theory, I think we've seen, I don't want to name names, but some data gathering operations have. The NSA leaks from all the way back to what's his name is now in Russia and Edward Snowden. And, you know, Facebook in the early days, the way anyone could just like log in as anyone and look at their stuff. There's carelessness and there's maliciousness and then there's governance. And yeah, I don't know what to say about it. Yeah, I think it's a hard line. It's something I'm sort of freshly thinking about. It's very hard to build and run these systems without feedback loops to let us see what's happening with some precision.

20:23Yeah, you need to understand the system, but how far does the system extend? I would love to be like, well, that's why the government makes laws and we can't blah, blah, blah. But I'm not going to try and do that with a straight face. Yeah. I don't know. Where is your head? I think the engineering mindset has the sort of data gathering and observability built into it. I think what becomes dangerous is when you take the engineering mindset into society, outside of machines, outside of rocks and metal. Yep. But, you know, engineers are smart people and they think that they can solve things that are outside the domain.

20:58So a lot of smart people do that. Oh, yeah. The smarter you are, the more you're like, well, this is just like that other thing I did. the Dunning-Kruger effect, which AI is making worse than ever. I mean, law really can do good here. I've been thinking recently about the effect of HIPAA. Before that, doctors regularly just told someone's parents or their partner or just told them things, sometimes instead of the person. That was incredible having laws and then making sure that you have access to your own. It was genius. We needed that. We had consequences, right? We had consequences. What do you think we should do, Ryan?

21:39Don't turn the mic on me. Come on. I think laws will be great. You know, we're finding the faults in the legal system right now. So maybe an engineering mindset on the legal system would be... I've been seeing all these news reports recently about lawyers using chat GPT and having filing briefs with errors in them and getting down. So I do think that the limitations in large language models are numerous and kind of baked in. And I think the future of software is determinism and non-determinism. And we use them both. But it's not magic. And I think that, I mean, I really wouldn't want to be a teacher right now.

22:20But I think the education system is some of where we're finding a lot of the edges on this stuff. And I think everybody is figuring out, like, oh, we have to change kind of everything. I had a conversation yesterday about SaaS and it's like a lot of this you can just write in-house now. But I think what's going to shift is that, you know, the SaaS that is indispensable, the stuff that's like, you know, a glorified database. Yeah, there's lots of types of software where you get a lot more value in having it be the same everywhere or in being able to share data. or SaaS is going to change, but not everyone wants to.

22:57Again, maintenance is what makes software costs what it costs. I'll briefly mention PaceLayers, which is the Stewart brand, which is the idea that this applies to all complex systems, whether it's society or technical systems or whatever, but there are some layers where innovation is great and things can move very fast. And there are some layers where you need it to move more slowly for stability. I think we run into a lot of trouble when we forget those and when we try to mingle them because it stifles innovation. If you're at a slow pace layer, it creates a lot of instability if you introduce a lot of change into a lower level.

23:35Like you said earlier in this conversation about boring software, innovation tokens, right? Yeah. You know, don't change too much. And be conscious of where you're changing it. Somewhere where it's easy to revert. Usually persistence is a lower pace layer. non-persistent stuff is much easier to like try and revert and move fast and break things because it doesn't it's wherever bits meet the world of atoms that things need to slow down a bit because whether it's humans and their interface or it's storing bits on disk it just it becomes a lot harder to revert from yeah speaking of move fast and break things i know you spent a lot of time at meta i did what would you say your biggest takeaway on moving fast and breaking things there was well by the time i got there we were all told that era had ended and i think it had okay it had i learned a lot of things at facebook none of them are technical none of them were things that i think i would have said i wanted to learn but they were all good lessons a lot of good people worked there but it was a one-man show it was the zuck show yeah what he said went to this day no one could eject him it's written right in yeah he has total control of that company.

24:47And for a long time, I think that that was a huge asset advantage because it allowed them to have some single-minded. So Zuck has great product sense. He's a smart man. And now I think we're really starting to see several years of compounding what the weaknesses is with the single player, the sort of monarchic model. No one can check him. Yeah. I mean, you're a CTO and a founder of a company. I wonder, like, how do you think about running the company differently? I probably am the opposite type of leader where I am not willing enough to be like, this is the way it goes. If it was easy, if there was a clear path to winning, you know, it's hard.

25:34Well, thank you for listening, everyone. I've been Ryan Donovan. I edit the blog, host the podcast at Stack Overflow. If you want to reach out with comments, questions, topics to cover, email me at podcast at stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn. I am Charity Majors, co-founder and CTO of honeycomb.io. I write at charity.wtf. I just put up a blog post called Confessions of an Unrepentant Slop Hater or something like that. We just finished posting a bunch of stuff about AI norms and values, which is on the Honeycomb blog. You're welcome to look up either.

26:09If you email me, I will probably not respond. Thank you for listening everyone, and we'll talk to you next time.

From the publisher

Ryan is joined by Charity Majors, co-founder and CTO at Honeycomb, for a conversation about applying the concept of immutable infrastructure to software development to treat code like cattle instead of pets, shifting the paradigm of system specifications to accommodate the rapid influx of AI-generated code, and navigating the complex ethical boundaries between system observability and human surveillance in modern technology ecosystems.

Episode notes:

Honeycomb is an observability platform that empowers engineering teams to visualize, analyze, and understand the behavior of their complex cloud systems. 

Connect with Charity on her blog at charity.wtf and read her team's posts on AI norms and values on the Honeycomb blog.

See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

More from The Stack Overflow Podcast

All episodes