Python, Go, Rust, TypeScript and AI with Armin Ronacher

25:27It might not always one-shot tasks, but it's pretty good. Mostly because the agent had access to context that's already in linear. Project plans, backlogs, issue descriptions, discussions. It's so useful. You can now delegate bug fixes, trivial work, or tedious tasks to the agent, then come back when it finishes. Linear sort of feels like ground zero. You've got to try it out. Head to linear.app.pagmatic.

25:49Armin Ronacher:Yeah, so we scoped out the email to be a fascinating space to work in. And so you can kind of imagine what the kind of problems are that you have when you work with emails in one form or another. and maybe a little bit of why we went into email is because one of the things that is very evident now of the biggest shift in at least my lifetime what's capable of computers is that natural language processing is now available at scale in many languages for a reasonable amount of money and that is a good reason to go into email because email is all about natural language and there's a lot of good data in it but it has been very hard to work with the data in the past.

26:31Armin Ronacher:If you read my blog, you know that my opinion on AI stuff for programming was incredibly negative up until February, March, where it slowly moved around. And then in May, I was like, holy shit, nothing is going to be the same anymore. And one big reason why I'm actually so bullish on this is because when you think into... And actually, it doesn't really work that well for Teams today, I think. Maybe there are ways, and I'm going to see as I'm scaling out this company how this is going to work at scale. but one of the reasons why I think it will actually be really good is because if I look over the years of all the stuff that I wish I would have been able to do I just didn't have the time for it and I had to make this explicit decision not to do it I have too many cases of this right like I remember we were reworking the grouping algorithm at Century.

27:20The grooving algorithm what is that?

27:22Armin Ronacher:It's like making sure that two errors which are very very similar are grouped into the same group. It's a data problem at scale, but it's also one with very specific data. And so we spent like three weeks building just a basic visualizer for ourselves to figure out if the new algorithm works better than the old one. I know for a fact that I get this tool in 30 minutes on the site from Claude. It's around 5 ,000 lines of code, and it looks better and has pretty UI and everything. So there are many projects that didn't go anywhere because the legwork needed to build yourself this bespoke tooling to even be able to pull this project off, like a migration system or something like this.

28:03Armin Ronacher:It's just too much, right? And so that's, I think, one of the big changes that I'm noticing for myself is that I have so many better tools around now that just make it much better to work with. I moved from Terraform to Pulumi for one of the services just to see how that goes. And I had Claude build me my perfect control system to get my logs and visualize what's going on in production. And I would never have done this before just because it wouldn't have worked. Another big question is like, how well does it work on a main code base, right? My co-founder is not a technical co-founder, but that doesn't stop him from using Claude or Codex now.

28:44Armin Ronacher:And it's actually great because it means that even someone who maybe you normally, like in certain stages of the company, there's a bunch of stuff where you wish you could do something but it actually requires maybe more coding skills than otherwise, right? And so there's an entire code base where I basically don't look at which actually validates a core part of the feature that we're working on and it's just vibes and it works. It's not pretty but it's not necessary because the whole point of it is figure out is this the experience that we want and this is great and then the thing is how much of the code that I'm writing which I feel like this is going to be a foundation of what we're doing, is at this point authentic.

29:27Armin Ronacher:And I guarantee you it's more than 80%. And there's very delivered code which is not written this way or where I really put a lot of effort into it. But most of the code that exists in this code base is not so important. It's a bunch of API endpoints. It's a bunch of open API specifications. It's a bunch of run-of-the-mill pieces of code they should look nice they should pass all the tests they should follow exactly the architecture that they want but if you were to compress them down and say like what's the actual information contained within them is very little right and so with the right approach you're actually very productive with these what made you so negative or like just like pretty negative up to around february or so the things that you were just pretty skeptical about and then why have you become so much more positive about like these ai coding tools or agentic coding right the biggest thing is that it it actually now starts doing work that i hated doing but i know i had to do so that is a like as an example yesterday i i really had to figure out why one of like the endpoints that i have work running there in production didn't work quite as it should and And it was not a problem of the error in the code.

30:46Armin Ronacher:It was a problem of AWS permissioning. And there was a chain of three things that I did wrong. That all resulted in it couldn't do what it was supposed to be doing. First, there was an IAM permissioning error. And then there was a whitelisting error in a different system. And then the final fuck-up was... I forgot what it was, I think. But the final fuck-up was also something with... Maybe that was actually a logic bug. But the point is, it was one error and one error and another error. And you couldn't see them all at once. And I was also busy doing something else. And I was looking at it like, I know I need to debug this.

31:22Armin Ronacher:But if I'm going to do this now, it's going to take me at least two hours. But it didn't matter because Claude was doing a lot of it. Like I still had to copy paste a bunch of things from the logs. But it knows so much because it has world knowledge access. It can combine a bunch of things together. and I could make progress on the thing I was doing while it was also fixing and debugging this production issue. And that is a significant improvement. The same thing with one of the things I hated the most was creating repo cases. But I know that every time I got a good repo case, I enjoyed it so much more, right?

32:00Armin Ronacher:And now I can say, Claude, make me the repo case for this. This is roughly what it looks like. Let's not try to fix it. I figure I was like, how do we get this one function that I can keep calling in a loop. That is what we're trying to figure out here, right? So I was not able to get that experience out of cursor, even with the authentic mode, prior to May, April. But that was really what changed my opinion on it. And then as I started going deeper and deeper into, okay, now that I know that I can do this and I can do this and I can do this, I also got incredibly adventurous in giving it more things to do.

32:41And now I have a new sense of where it could go just by giving it the things I don't want to do, but staying in control of the things I do want to do.

32:53Armin Ronacher:And that was the big shift for me, was recognizing that it can do that in a way. And so far, what are you seeing in what these agentic tools change about software engineering? And what are the things that don't change? Especially, you know, you've been writing code for like, what, 20 years or so. So system architecture and like overall complexity and like how you build things in a way that something stays maintainable as it scales into a larger and larger company. There is a lot of experience in it. and I also didn't have all of that experience at any point in time. So I feel like this is one of the things where I really like as time went on, I learned a lot more, which gives me a benefit over my prior self that didn't have that.

33:42Armin Ronacher:That is going to continue to be true. Now, there's an argument to be made that maybe Claude is like systems like this will help you and maybe can even bootstrap these sort of things around it. If you fully delegate everything that you're doing to a machine, then the person that doesn't do that has an edge on you. Right? Because there's going to be some innovation that is not in these things yet. Right? So right now what these systems spit out is sort of our best understanding of how stuff works up to this point. Right? And so having something that maybe at scale is not in these corpuses yet is where some of the edge comes from.

34:19Armin Ronacher:It also creates a better culture in a company if you're not completely delegating everything to a machine. Right? Because what's the point if it's just a bunch of like this? This is one of the reasons I want to hire more people now. It's just because Claude is not a human. And it's just that there's an energy to a well-running company and to motivated people that just a computer can't replace. So I think that is a big part that sticks with you as an engineer is like understanding how to create the systems, understanding when the code that it produces is more right versus more wrong. So I don't think much of this has changed.

34:57Armin Ronacher:I feel the same programming with Claude, even if I physically write less code. It's just that, I don't know, the physical act of punching the keyboard is not quite the same anymore. What is different though? What has changed with it? I think the cost of building your custom tools went down. The ability to make better decisions now for me is heavily improved. In parts, it is because it Googles very well. It searches the internet very well. If I now have a problem of a type that I didn't have before, it can teach me this problem. This is one of the things I use it a lot with. It's like, I don't actually want to get the answer.

35:43Armin Ronacher:I also want the answer, but I also want to understand why that's the answer. and it can do that and it can put it into the perspective of it. I'm not very good with a lot of very complex math and I found that every once in a while I need to do that and having chatGPD dumb it down for me helps me a lot. There's seemingly, if a problem is presented in a certain way, it works very well for me. If the problem is presented in a different way, particularly in a very abstract way in which you have it in many papers, I do very I'm not nearly as good as I should be, I think, for what I'm trying to do. But ChatGPT helps me grasp it quicker, right?

36:26Armin Ronacher:So I think that's a huge change. The other thing that definitely has changed, and I even wrote about this recently, is that it brings a whole bunch of people into programming that didn't do this before. Yeah. I wrote about this before, but I had a four-hour conversation with an air traffic controller on a train who talks about how he uses ChatGPT and how he can even do some programming with it. It's like, that's the programmer now. He has a very specific problem. He wouldn't have programmed before, but because there's codecs in the ChatGPT subscription, he's able to solve some problems that he wasn't able to solve before.

37:02Armin Ronacher:And that is a huge unlock, right? We are going to have many more programmers just because as a byproduct of asking a question, ChatGPT is programming. And a human is going to keep running that code and maybe it wants to do some changes to it. One of the side effects of Flask being so popular is that it has been in university courses. It has been in onboarding for new programmers in boot camps and things like that. So I've been exposed in one way or another through how long it takes for someone to learn programming. That's not a thing that you do in a day. So for you to be a productive programmer that's capable of splitting out the program, that's a couple of months of work.

37:39Armin Ronacher:And now someone can press a button and get something. And then maybe they enjoy programming for that, right? Maybe there's like, well, now I see the effect of having produced something. Like maybe you didn't do all the things, but it's still the same gratification you get from it. And then maybe that's a way for you to, for people that wouldn't have gone into programming, to actually go into programming because the onboarding has been so much easier. We talked a lot about programming languages and you have very specific, you know, you have your opinions on them. but with AI and cloud code codecs and all these more and more capable agents being here how important how do you think the importance of programming languages will change?

38:19Do you think it'll be limited to a few people or just fewer and fewer people like yourself or others who actually have spent a lot of time understanding the pros and cons of programming languages thinking well beyond because it sounds like you can just tell AI to solve it and whatever, give it an existing code base it'll just do it, right? So there is an argument to be made that programming languages might not be that important anymore. What is your take?

38:42Armin Ronacher:Well, I think I will be consistent with the prior thing that I said, which is I think I have a lot of respect for programming languages because of the trade-offs that are in there. And with AI now, I think you might have to start looking into different kind of trade-offs. The reason, another reason why I ended up with Go in my startup is because I just noticed how good Go code scales for AI because the abstractions are very thin. It understands the code better. So clearly there's some, I did the measurement, right? It's like I made it write a certain type of program in different languages 10 times and see like how often did it pass?

39:18Armin Ronacher:And I just noticed that it did so much better on Go than it did on Python and much better than at Rust. Clearly languages matter because the quality of what the agent is going to do will matter. And then is any of the languages they have right now, is that the perfect language for both a human and a computer combined work environment? I don't know. Maybe we found the pinnacle of programming languages, but maybe that's also exactly the moment in time where someone will comment and say, like, hey, I actually have a brilliant idea of what the better tradeoff is here. Because the cost of doing this kind of stuff is going down with AI, but the cost of this and this stuff is going up because of AI.

39:55Armin Ronacher:So let me rebalance this in a different way. so actually I think that I think that program language will continue to matter a lot particularly the trade-offs that the program language implies on the runtime environment matters a lot more so I don't think that's going to change I think it's just going to everything accelerates, everything is going so much quicker right now or it feels to go so much quicker because the fundamentals are shifting around and this is not just with now everybody wants to build all the new things it's also like you feel like if you're not working all the time now, you're missing out on some of the changes.

40:31Armin Ronacher:And I don't think that's just on the product side. I think it's on the tech side too. I'm pretty sure there are a bunch of people who are feeling like, I want to be the first person to build the programming language that works really well for agents and humans. But it's not just one person. Probably a lot of them who are looking into this at the moment. It just feels so... Like there is, in fact, a moment right now to build the perfect language or a better language that works in that environment because we are probably not going to get rid of authentic coding. And what is the likelihood that all of the languages that we have right now are the best ones?

41:07Armin Ronacher:Probably not that high, right? So it will be the same and different in many ways. And one thing not to be underestimated is that the likelihood that computers are just going to program and no human is going to be in a loop, I don't think is very high. I think the human will stay in the loop longer than we want and in many more cases than we want. And so you cannot go down to say, like, well, the optimal output is, right? A bunch of assembly code because then nobody can review this anymore, right? And so if anything, the trade-offs will have to be like a higher level so that reviewing gets easier.

41:39Also, don't forget that the thing with assembly, right? Like one of the reasons we have higher language languages is assembly needs to be different for every architecture type or every CPU type, which is why, you know, like Java was so popular in the early 90s. It ran on both Macs.

41:57Armin Ronacher:But if I can't just have the AI ported to my auto-operating system, they could make the argument that maybe that is not needed anymore. For my support, I don't think that's the opinion that I hold. But it changes the cost of some of those things, right? Yeah. But you mentioned an interesting thing, which is how everyone seems to just be working all the time, which is this interesting contraction, how AI or these agents you can kick off and you could, in theory, go to sleep and have it just run in loops for you, and so you could work less. But, in fact, we're seeing working more. And we now have this thing, which is very amusing to observe, that AI startups in particular are increasingly demanding or advertising this 9.96 from 9 a.m.

42:39to 9 p.m. six days a week, especially in NSF. They're putting it in the job adverse. They're posting it probably online that in the office at midnight or after midnight people are still there and working hard uh you you share some thoughts on this as well on your blog it caused quite a bit of stir but what is your take on on this because you did say you know like there is a lot of energy right now there's a feeling of like not wanting to miss out clearly that's why why these people are doing it yeah so a handful of thoughts

43:11Armin Ronacher:one of which is i actually have to credit peter steinberger quite a lot of uh pushing me into agent decoding because he uh he worked in a company called pspdf kit which is sold and then i don't want to tell his story but long story short at one point he was kind of stopped programming and then he fell and it's like hey i found this this computer which programs for me and he he said like he doesn't sleep anymore he's like i'm so addicted to it you should try this in a way like he didn't quite say it like this but like i'm skeptical right but the computer that does your work you cannot sleep because of the that's kind of so amusing yeah it's like i i realized like it does something with your brain if you if you do this a trendy coding thing because like in the beginning i felt like any minute i'm not having it running and doing something i'm just wasting time right now it's like it was a huge part about it's like it's almost like a drug right it it has this instant gratification of something happened and and you can kick it off again and kick it off again.

44:11Armin Ronacher:It's like this, it is slot machine, right? And so I think I attribute some of it to that. And it took me a while actually to come out of this way of thinking because it's not very sustainable. Like there were a bunch of nights where I just, I did through the night with a handy coding. Not because I felt like I'm incredibly productive on something using from a startup. Just like, this is blowing my mind. I just, just one more, just one more prompt, one more prompt. So I think like Like to some degree, because if you're so AI close, that probably also contributes. The other one is I think everybody sees the change and wants to do as much as possible.

44:48Armin Ronacher:I think the tricky bit with the 9 and 6 thing is that 9 and 6 defines a very, very specific kind of work regimen, which is you work 12 hours a day and you basically don't have a weekend. I never did something like that, even though I probably worked sometimes 80 hour weeks. and the reason for this is that i know that i cannot be productive all the time if i if i work late at night i will shit in the morning i have three kids i have a wife i i family is very important for me it's the most important thing so i'll always arrange my work around that you can work with intensity but you can still um like do it in a way that is not crazy hours at crazy times where it shows that you're optimizing to the optimal output that you can produce somehow.

45:38Armin Ronacher:And I also want to say there's a huge difference between if you are the founder or if you're a very, very early engineer with a really good compensation system where equity means something to you from most of the people who join a company later on or where the company has no realistic path of their equity being worth something. There's absolutely no point in putting a ton of effort into something where only one person benefits from. And there's a lot of subtlety to all of this. And what annoys me with the nine and six point in particular is that they picked that word, which if you go back to why that naming even exists, is because of basically there's no subtlety to it.

46:26Armin Ronacher:It's like you work 12 hours a day, six days a week, and you give up what is otherwise in your life. And I don't think it's a trade-off that anyone should make at any point in time because there's more to life than working in a company. And that is not the thing that you should be putting on your flag, which has such a negative connotation because of exactly what it did. Well, people literally killed themselves over it, right? And if you want to be a high-intensity work environment, then say that. But be more... Because in practical terms, you cannot be a company that runs like that. You can only be a company where high-energy people work and they build themselves up.

47:15Armin Ronacher:But there's a limit to which you can do it. And you have to be transparent about what the cost of this is. And I've been in the industry long enough that I've seen many, many, many bad outcomes. And in particular, through my community, a lot of people even went down like psychosis and schizophrenia and a bunch of things where your likelihood of going down that also increases during certain years with unhealthy behavior. Now, having worked at Accenture, which is all about error handling or at core, you've probably seen tons of stats on most common errors that are coming up in different servers.

47:53What have you learned about error handling and how to build better software with fewer errors from a developer's perspective?

48:01Armin Ronacher:So the biggest thing that I learned, and ironically, hasn't really dramatically shifted, despite the existence of tools like Sentry and their popularity, is that many of the ways in which errors are worked with just doesn't carry enough information or only carries useful information for debugging in debug builds. and that's problematic because the most interesting errors don't happen in debug, right? Now, maybe this is changing a little bit with AI because like the cost of producing a reproducer from not perfect information might go down. So maybe there's a counter move to that, but I really felt that as a language designer, as a VM creator, you should put a lot more emphasis on making sure that the errors can carry really useful information rather cheaply, even in reduce and in production runs.

49:06Armin Ronacher:And so not a lot of ecosystems don't do that, which is very sad. And I think that the fact that Python had such great introspectability without further, like Python is a slow language. So if you already have all your debugging tools at it, it doesn't get significantly slower. That actually was the reason Centri existed. Because Centri was able to show local variables for Python. It had all this rich, powerful data that for many languages, even today, we're not able to do because the runtime doesn't support it. Or the runtime effect of doing that is too high. And so that's the thing that I learned the most.

49:44Armin Ronacher:Your experience of debugging issues is so much better if you have rich information errors. And unfortunately, both languages, language creators, runtimes often neglect errors. So they don't carry the right information. And the application and library developers often don't think about errors at all or they capture them down in the wrong places and then the useful stack trace is gone, all that sort of stuff. Because you think they're exceptional, they're not happening all that often. But unfortunately, every time they did happen, you know, you didn't have enough data. So just in general, the design of errors is completely, it's not where it should be given how important they are.

50:30So that's one thing. But Sentry works across a bunch of different languages. Almost all languages, right, these days, or at least the ones that are widely used. Have you seen interesting patterns between the types of errors or the frequency of errors of using languages? which is basically I'm trying to get to like, how important is it to choose your language in terms of, you know, like having more correct programs?

50:56Armin Ronacher:Well, different languages crash in different ways and not just because of the language, but also what you build with it. And so, for instance, JavaScript at scale, like if you look at a lot of websites, errors all the time. The percentage of errors in JavaScript that actually are like meaningful is very, very low. Yeah, like whenever I open up my deep developer tools and then it shows you the errors and warnings and like every single website or almost every. And that makes sense because very few errors that you can actually cause in JavaScript and crash your browser tab, for instance. Right, so it's like on error resume next.

51:39Armin Ronacher:You're hobbling along. Like something is broken but doesn't really like, but like the existence of an error in a console log doesn't imply that the website is broken, but for instance, now the event listener might not fire anymore, so the click is a dead click, which is something that essentially needed a session replay product to find, because the actual error that you would find in century was insufficiently linked to user not being advanced on the page. On the other hand, if you have a computer game, it's written in C++ and it crashes, your session is over. Right? And so the comparative fewer crashes in computer games compared to all the nonsense going in JavaScript.

52:16Armin Ronacher:But when they do happen, they're much more meaningful. And so it's very hard to say error rates in different languages and so forth because the impact that these errors have is very, very different. So C++ code crashes in computer games, shockingly little. The amount of traffic that century gets from these is very, very low. But the usefulness of each individual crash report in C++ is so much higher. So it's a very complex topic at scale to say, how does something error because it really depends on what does it take down. Obviously, there are certain types of errors that if you see them long enough, you feel like they really shouldn't happen.

52:53Armin Ronacher:To some degree, there was a large-scale realization in JavaScript ecosystem that type checkers could get rid of a whole bunch of class of errors because at the very least you have to explicitly check if this thing is nullable or not. But I also didn't ever get the feeling that the adoption of TypeScript would dramatically change anything about like sentry's javascript error rates if anything like none of that adoption had any meaningful impact on really how crashy yeah well because because what what what i would have expected right like we know and i think it's pretty commonly assumed i'm not sure if it's proven that type safe languages will reduce certain kind of more obvious errors.

53:36TypeScript does this with JavaScript. I mean, it has a compiler, but in the end, you do get a check before it all gets compiled to JavaScript. So we should have fewer. That's what I think. So you're saying that you didn't really see too much change.

53:51Armin Ronacher:No, I think if there was an impact, it's unmeasurable. And of course, you have other sort of frequency functions that might happen that offset. Maybe whatever improvement that you have on catching some of those, maybe that sort of makes you just more adventurous to build more complex code in comparison, right? A lot of the time of Sentry was the Zerb era of crazy complexity. And so then many of the errors were related to not like, was this nullable or not? But now all of a sudden, it's a microservice that I'm talking to, and they have misaligned versions. And my type check actually didn't help me at all because someone decided that on a network layer, it's now null anyways, right?

54:38Armin Ronacher:So the increased complexity of many of the software that's actually dealt with over the years, I think you could have probably measured that. Especially I remember the adoption of more and more complex things in a React ecosystem. just dramatically increased the types of errors. Like hydration errors were not a thing for many years. And then all of a sudden, there's a whole class of new errors coming in because now all of a sudden, the view stability between initial server render and the JavaScript dynamically loaded on time does. That was a whole class of errors that didn't exist before and all of a sudden does exist, right?

55:16Armin Ronacher:So it's very hard to measure these things just because of how we change and how much more complex our apps are. Yeah, so you're saying that's true before React and hydration errors are specific to React, but that whole error category just didn't exist. And then as React spread, they're kind of pretty common in React. So now technically you get more errors. It turns out to be a very safe business to be in errors because they're not really going down. Yeah, well, and then I think we can extrapolate it, obviously, with AI tools typically using the most popular frameworks or the ones that has the most training data, which is the most popular one.

55:54So we'll likely see an explosion of code and deployments. So, yeah, it's probably a good business. but having been in the errors business how did you change your your your or at the teams the team at century how did you change your approach to building software with errors in mind were you more aware of it did anything change at all because it's interesting you know you you've had such good insight into like all the i'm gonna say mistakes they might not be mistakes but we can say you know error is generally not a positive thing but you've seen all these things come through so did you change how we develop software?

56:31Armin Ronacher:You would think that working in an observability company makes it great at building observable products. And the reality is that there is a reason errors are an afterthought. It's like you almost have to force yourself to take care of this 5 % case or 1 % case. And that's a continued problem. Every pull request in a way you and I are like, okay, now it passes. So we have dealt with this. Like, are you going to put the extra time in to also make sure that even if it like now fails, that it reports the error correctly? I don't think that sentries code is more correct for error reporting than any random other code out there is because the act of taking care of good error reporting is a deliberate act.

57:19Armin Ronacher:The act of like good logging is a deliberate act. The act of good metric supporting is a deliberate act. And you will train that muscle a little bit over time but i think like you maybe get 50 better 100 better at it which is still very very low we should get like a thousand percent better like it's like that that's where we would see that like the improvements like if if everything like if 90 of errors are really like error cases are really well reported like like that would be really good but like that's a really massive lift and you would only really get this lift if you stop programming like we are and maybe the languages themselves have, like pulling this into every stack frame, into every sort of corner case.

57:59Armin Ronacher:So I think that the real lift would come if a bunch of people that now build these observability companies say like, okay, now that we know what our problem is, which is data collection in the first place, how do we build a programming environment that makes this native, right? For many, many years, the biggest challenge for just bringing awareness of what we're doing was context locals. What is context local? There's an excellent talk by Ron Pressler, who wrote the, I think, the original implementation of virtual threads for Java, who later worked on the Loom project to bring virtual threads straight into Java.

58:44Armin Ronacher:Basically made a strong argument for the stacked frame. Like, what's so great about stack frames? Like, stack frames are amazing because they carry information basically implicitly with it. And he sort of demonstrated what you give up if you move to async await, where you basically chain together a bunch of promises to each other. And the immediate problem that we have if you start moving towards promises is that now it's very hard to carry through information through all of the potential paths that this promise chain would take you. So for instance, if you want to say like I have a correlation ID, which should be attached to every single log event, it should be attached to every single error event that might happen as a result of this original thing I'm doing.

59:26Armin Ronacher:With stack frames, super trivial. It's sitting somewhere in the stack. I will not lose it. I can walk up the stack and find it again. But with promises, now I need to have this chain together. And so context locals is what is called in some languages. In.NET, it's called the execution context. it's basically a hidden piece of almost like a threat local but a much more narrow scoped one which flows with the logical flow of execution. And why is this necessary? Because if you do something like open telemetry if you do tracing if you do things like you want to have a correlation ID you want to have it show up everywhere, right?

1:00:01Armin Ronacher:And for many years, a century it was like just talking to people like hey, you know what would be really great to have? Context locals everywhere. And JavaScript, I think in a browser still doesn't have them. in the backend, it eventually got domains initially, then it got async hooks, and nowadays it's called async local storage. It sounds like to me, like as you said, like as an engineering team, let's say even at Sentry, you kind of, your focus is to get the thing done, you know, you test that it works correctly. And then when it comes to like errors, as you said, it's kind of usually an afterthought, either when it blows up or, you know, if you've got a lot of time or if you really need to make it sure that it's going to work correctly.

1:00:40But with languages, programming languages, it almost sounds ironically that in many languages, it's an afterthought. Because as you said, if you thought it through better, saying, hey, at some point the program will crash or there will be an error, what can I do? And there's a context local thing that you talked about.

1:00:56Armin Ronacher:I don't think it's an afterthought. I think it's deliberately not done. Because you have to imagine there are different kind of forces in a programming language that say, like, what should the language do? And some of them are very clearly at odds with each other. So what's the problem with context local? They make every call slower, right? So now it's like all the people that want a faster language, they're like, well, now you've got to make a really strong argument why you should be doing this. I remember one of the biggest fights over the years that was hidden behind the scenes was on native platforms, compilers at one point made the decision that we can get one extra register by basically using the stack register.

1:01:37Armin Ronacher:You would basically give up the last register to recover the stack frame for one extra register for other users. And then you would have to use a very complex Dwarf unwinding system to recover this, right? So you basically say, like, we're moving some of the complexity of being able to just add runtime, walk through the stack, to doing it delayed through a very complicated binary program, which is embedded in this debug information file. But what do you lose for this? Well, you lose runtime profiling capabilities because that's too slow to do at scale. So I can't do an in-process profiler, which turns out to be incredibly useful because you will find information you wouldn't find through tracing, right?

1:02:20Armin Ronacher:You can do like, basically, you can run a sampling profiler. It also made it incredibly hard for Sentry to do proper stack reports for native code because I might not be able to recover the debug symbol because it's a binary driver from someone. and nobody ever gave me the DLLs. Yeah, you don't have a source code or the mappings. I know like Facebook, for instance, Facebook's app secretly uploaded Android system symbols all the time. Like your device were randomly assigned if it should upload it or not. And they were harvesting all the system symbols to increase the quality of the error reporting.

1:02:55Armin Ronacher:Like if your Facebook scale can do that, like Sentry couldn't do that. Like we can't just magically have every Sentry SDK just upload a bunch of files. It's just, that would just destroy user trust. but eventually I think that only took place two years ago someone at I think it was Debian or Red Hat was like well this is nonsense we're going to change this now we're going to bring the stack pointer back and it was a fight it was a fight to do it and it was like well we bench all of it we're just losing 5 % except for Python where the Python interpreter for whatever reason got like 20 % slower so they didn't do it for Python but it was one flag right it's one flag but what matters is that flag like I said, by default to everything, right?

1:03:36Armin Ronacher:So it should be on by default. But this 5 % of performance difference, which was assumed to be more than that, there was a year-long pressuring, right? So I think it's hard to do because you have these different interests in it. And some people want the performance and some people want the debuggability. And they might live in completely different worlds. Like maybe this one person that really cares about performance just never has to look at any crashes because that's not the area of responsibility. And then you have someone who maintains a fleet of complex services and every one of them has a small percentage of crashing and often doesn't run the code that they have.

1:04:14Armin Ronacher:And then once they have to crash, they don't have the data that they need. This is so interesting because it reminds you of how difficult it must be to design a language. Because again, you have so many users and as you say, the person who just had a crash, They actually want ideally the language support to get that back in a very easy way. But now it will by default slow it down either through performance or through higher memory usage. Like you cannot have a free lunch, right? Yeah, it looks like the lunch comes with some price. But look, I have a lot more respect now, I think, for language designers just because of, like especially with startup programming, you have this naive idea of like it has to work this way, right?

1:04:54Armin Ronacher:And then over time, you realize, oh, there are all these trade-offs you have to make. And some of them are really hard to quantify. And you might even have to make some very unpopular decisions. And there is no real right or wrong. There's just a lot of trade-offs. And maybe this is also why we have so many languages to some degree. But it is very fascinating to think about what some of these seemingly little defaults have in terms of what you can do with it. And then maybe what kind of people are going to pick your language. I thought Python was the most amazing language ever for building web services because I could inspect every process and it didn't make it any slower.

1:05:31Armin Ronacher:But now I know that's also why Python was slow to begin with, right? What are some things that you've learned being an early engineer, almost like a founding engineer at a startup? And what is advice that you would give to an engineer joining a startup, a fast growing startup right now, early on? Well, I can give the advice from someone who would have joined the company 10 years ago, right? Look, the situation changes all the time, I think. But my life in many ways is a very linear path. Where it's like, there's an opportunity, I make a decision if I take it or not. It's like the secretary problem, but I'm kind of taking not too many attempts at it.

1:06:09Armin Ronacher:Right? Which is like, I just want to, like, the passing bar for me is, do I think that this is great? Do I want to work on it? And then maybe I take the opportunity to work on it. And for me in particular, I'm shaped like a person that is actually not a great employee because I don't necessarily go and fit a particular mold. My titles at Century in many ways over the years were, I didn't even associate with them in a way. At one point it said, I don't know, software engineer on it, but I was responsible for an office here with people underneath where I sometimes felt like I'm a payroll provider.

1:06:51Armin Ronacher:There were so many other things in it because a small startup has this many, now someone needs to deal with this stuff too. You need to hire someone to even just bring them things like furniture into the place. If you join a company early, everything is in flux. Nothing is well-defined. And some people like that. Some people don't like that. And I think you have to be just willing to experience new things. If you start at a scrappy startup that particularly maybe comes from someone who hasn't founded a company before, hasn't been there when a company sort of grew from small to big. Because even now for me, I just realize again, oh, I kind of forgot I need to do all of this and this again.

1:07:34Armin Ronacher:but there are people that like that and people don't like that. And you just have to be aware of the consequence of this. You have to path the way for yourself somehow, either by being okay with just doing whatever comes up or being incredibly ambitious and liberty taking certain paths towards where you want to go, even if that might temporarily set you back in one form or another. And now you're the one who started a startup as a software engineer. You founded your startup with your co-founder. So far, how is it going? And how is it different to what you've expected, especially that you were on the other side, right?

1:08:13You were an early employee inside a startup. You've seen a lot of things. Is it kind of like, you know, exactly as you expected or is it just a different feeling?

1:08:21Armin Ronacher:In some ways, I postponed the founding of the company many times, right? In my mind, at least, my recollection of early century was like well there could have been a path where we didn't do century we did something else right um and then after almost like four or five years already like i remember like when my first um like where all my shares wested i was like hmm is is there a path where because there was also was coming out at one point where they took next chairs and built an entire business around like hey is there is there a path where you could take actually something like flask and and build a think around it.

1:08:59Armin Ronacher:So in some ways, there was like a multi-year preparation for me. So now it doesn't feel like, oh, it's actually different. It's just like, okay, that's the next logical step. I wouldn't call them missteps, but knowing some of the decisions we have made at Century, I can now look at them and sort of look them from a different perspective. I have different opinions now about the equity, about the consequences of equity, how to structure certain things. Like this time around, basically, I didn't start with, here's an idea for like a product I want to build. The reason I ended up starting this company with a co-founder is we actually sat down and said, hey, what do we want a company to be like?

1:09:36Armin Ronacher:Why are we even in here for it? We already know that we're the kind of people that want to start a company, but it's not an end on itself. Something should come out of it. And so I didn't do that last time. I just rolled into it. Now you can overthink it, obviously, but it's also, there were many moments over the years a century where I felt like, well, actually it would have been great for us to think about some of those things. So now I'm thinking about them in the beginning. The experience primarily puts you there. I don't think you can sort of start out and say, I'm going to do this. Obviously, you can read a bunch of stories of people that started startups and then you're listening to other podcasts and everything and you hopefully get a better sense of it and maybe to some degree overprepared.

1:10:18Armin Ronacher:But having gone through it yourself versus just hearing from someone. I read a lot of stuff about startups, even 2015. I was sort of addicted to that thing. And still it feels very different now doing it versus if I would have done it five years ago, just because of my own inner experience, sort of like how I feel about all of this, which when I have a level of urgency versus where I just sit it out, the experience kind of puts you in a different position than I would have done otherwise. I don't know if that's helpful, but that's I think mostly how I feel. Well, I mean, it sounds like you can get all the opinions, read about it, listen to stories, but it'll be different when you do it in the end.

1:11:02And it's hard to tell what ways until you do it.

1:11:05Armin Ronacher:Yeah, and it worked for some people, but it wouldn't have worked for me. Like, no amount of reading of anything would have sufficiently prepared me. Some people, it might, right? Like, particularly if you're well-educated, like, a person grew up with the right values to have this intrinsically sort of put on to you. But that was not my education, right? My education was, like, just standard schooling, bunch of university. And then there's a whole world that I haven't been exposed to that sort of I had to learn in one form or another. and then just to close with some rapid questions so i'll just ask and then you tell me what comes up of the programming languages which one is your favorite and i'm really interested to hear it's python and two answers for why uh one because it gave me the career that i had and so there's obviously a lot of emotional attachment to it but also it is it is a bad programming language for many, many aspects of like misdesigned and whatever, but it is incredibly pragmatic.

1:12:05Armin Ronacher:And I just liked that. And I kind of want to point towards many of the things that Cal Henderson did over the years. He was the CTO at Flickr originally and then Slack. PHP guy, right? But the pragmatism in which he used PHP was in many ways a pragmatism, which I always like building product with Python. and it's like you don't really care if it's good or bad and that sounds like what you can do with it. And I could do a lot with Python and I appreciate it for this. And then what's a tool that you love using and what does it do? I will answer this as an auto-programmer. A tool I love using is a screwdriver.

1:12:42Armin Ronacher:It screws. But the reason why I just love it so much, I think it's one of the things that I learned over the years. I never really had a good electric screwdriver and then when we bought our apartment, I just bought really good tools, including a screwdriver. And it has increased my willingness to drill holes and assemble furniture and everything. And that by far is probably the most favorite thing that I have now. It's just really, really good, well-manufactured screwdriver. I wonder if there's a metaphor there that will apply to development as well. Like, you know, when you have good tools, you're actually a lot more motivated and willing and you're more adventurous as well.

1:13:28Well, thank you for this chat. This was really fun.

1:13:32Armin Ronacher:It was really nice talking about it. One of the most interesting parts about talking with Armin was how he went from being very negative about AI six months ago, and now he's building a startup with AI agents like Cloud Code and OpenAI Codex. I see Armin as a very pragmatic engineer, and if he went through this much of a change after pushing himself to use these tools more, it tells me that it's worth giving AI coding tools some time and see how they work for you. For more observations on AI tooling trends, check out Deep Dice into Pragmatic Engineer, which are linked in the show notes. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube.

1:14:06A special thank you if you also leave a rating for the show. Thanks, and see you in the next one.