In short
Podcast Episode Notes: Build Software That Lasts! (Interview with Bert Hubert)
Episode Details
- Podcast Title: The Changelog: Software Development, Open Source
- Episode Title: Build Software That Lasts! (Interview)
- Guest: Bert Hubert
- Host: Jared Santo
- Description: Bert Hubert shares insights from over 30 years in the software industry and discusses what it takes to build software for the long term.
Key Concepts & Discussions
Introduction
- Bert Hubert is introduced as a seasoned software developer with a diverse background, including founding PowerDNS and creating a parliamentary monitoring system.
- The focus of the episode is on long-term software development and the principles that lead to creating lasting software.
Early Career Anecdote
- Bert shares a story from his early career where he accidentally took down an internet service provider by improperly managing server tasks. This experience taught him about the importance of professionalizing processes and understanding system redundancy.
Lessons on Redundancy & System Resilience
- Bert emphasizes the significance of testing redundancy in systems. Just claiming redundancy isn't enough; practical tests are necessary to ensure reliability.
Building a Parliamentary Monitoring System
- Bert built a parliamentary monitoring system to track important legislative documents and discussions. His system is notable for being accessible to people with disabilities, showcasing the importance of user-centric design.
The Challenge of Long-Term Software Development
- Bert discusses the shifting landscape of software development, especially with many dependencies in modern projects. He highlights several pitfalls:
- Dependency Drift: Changes in dependencies can lead to silent behavior changes in software.
- Version Changes: Major version upgrades can necessitate significant rewrites.
- Abandonment: Dependencies may be abandoned or monetized, impacting their availability.
- Conflict: Conflicting dependency requirements can create maintenance headaches.
The Importance of Simplicity
- A key takeaway is the need to keep software simple. Bert and other industry veterans agree that complexity often leads to maintenance challenges down the line.
Strategies for Selecting Dependencies
- Bert suggests thorough scrutiny of dependencies and provides guidelines:
- Assess the community and support around a dependency.
- Evaluate the long-term sustainability and funding of the project.
- Engage with the project actively to determine responsiveness and support.
Dependency Management and AI
- Discusses the potential of AI in managing dependencies, suggesting that AI could help automate the scrutiny of dependencies and provide insights about their reliability.
Continuous Learning and Adaptation
- Bert stresses the importance of staying relevant by adapting to new technologies while not getting too attached to legacy systems.
- He encourages developers to find a balance between embracing new technologies and recognizing the value of longstanding, proven tools.
Conclusion
- The episode wraps up with a call for listeners to consider their own dependency choices and maintain simplicity in software design.
- Bert encourages developers to engage with their communities, share knowledge, and continue learning.
Key Takeaways
- Test Redundancy: Always validate claims of redundancy in your system.
- Simplicity is Key: Avoid unnecessary complexity in software design.
- Stay Engaged: Actively participate in the communities around your dependencies.
- Continuous Improvement: Regularly assess and refactor your systems to keep them manageable and efficient.
- Balance New and Old: Embrace new technologies, but don't discard the value of proven approaches.
Additional Resources
- Connect with Bert Hubert:
- Bert is active on Mastodon and is launching a newsletter for updates.
- Changelog Community: Join discussions on software development at [changelog.com/community](https://changelog.com/community).
---
These notes summarize the topics discussed in the episode, emphasizing the lessons from Bert Hubert's experiences and insights into long-term software development.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:05I'm Jared Santo and you're listening to the change log where each and every week we have conversations with the hackers, the leaders, and the innovators of the software world. We pick their brains, we learn from their mistakes, we get inspired by their accomplishments, and we have a lot of fun along the way. Today we are joined by Barrett Hubert. After over 30 years in the software industry, Barrett has experienced a lot. He founded PowerDNS, published articles for places like IETF and IEEE, and built his own parliament monitoring system. That just scratches the surface. But recently, Bert wrote about what it takes to build software for the long term.
0:48We dig in. But first, a quick mention of our partners at Fly.io, the public cloud built for developers who ship just like us. Yes, Fly is the home of changelog.com. Check them out at fly.io and tell them changelog sent you. Okay, Bert Hubert on the changelog. Let's do this.
1:14Well, friends, before the show, I'm here with my good friend, David Hsu over at Retool. Now, David, I've known about Retool for a very long time. You've been working with us for many, many years. And speaking of many, many years, Brex is one of your oldest customers. You've been in business almost seven years. I think they've been a customer of yours for almost all those seven years to my knowledge. But share the story. What do you do for Brex? How does Brex leverage Retool? And why have they stayed with you all these years? So what's really interesting about Brex is that they are an extremely operational heavy company.
1:47And so for them, the quality of the internal tools is so important. Because you can imagine, they have to deal with fraud, they have to deal with underwriting, they have to deal with so many problems, basically. They have a giant team internally, basically just using internal tools day in and day out. So they have a very high bar for internal tools. And when they first started, we were in the same YC batch, actually. They were both at winter 17 and they were, yeah, I think maybe customer number five or something like that for us. I think DoorDash was a little bit before them, but they were pretty early.
2:15And the problem they had was they had so many internal tools they needed to go and build, but not enough time or engineers to go build all of them. And even if they did have the time or engineers, they wanted their engineers focused on building external physics software because that is what would drive the business forward. Brex mobile app, for example, is awesome. The Brex website, for example, is awesome. the best expense flow, all really, you know, really great external things in software. So they wanted their engineers focused on that as opposed to building internal CRUD UIs. And so that's why they came to us.
2:45And it was honestly a wonderful partnership, but it has been for seven, eight years now. Today, I think Brex has probably around a thousand Retool apps they use in production, I want to say every week, which is awesome. And their whole business effectively runs now on Retool. And we are so, so privileged to be a part of their journey. And to me, I think what's really cool about all this is that we've managed to allow them to move so fast. So whether it's launching new product lines, whether it's responding to customers faster, whatever it is, if they need an app for that, they can get an app for it in a day, which is a lot better than, you know, in six months or a year, for example, having to schlep through spreadsheets, et cetera.
3:24So I'm really, really proud of our partnership with Brex. Okay, Retool is the best way to build, maintain, and deploy internal software, seamlessly connected databases, build with elegant components, and customize with code. Accelerate mundane tasks and free up time for the work that really matters for you and your team. Learn more at retool.com. Start for free. Book a demo. Again, retool.com.
4:03We are here with Bert Hubert, a geeky entrepreneur from the Netherlands with a 30-year track record in commercial and open source software. Bert, welcome to The Changelog. Thank you for having me. Happy to have you. Excited to talk about what you're up to these days and what you've been up to historically. 30 years, that's a long time. I read that you landed your first job hacking a cable internet provider. Do you want to tell that story? It was a fun story. So we lived in this university town and we were quite late to get the cable modem. And it may be interesting to know that in Europe, in many places, if you had dial-up internet, it was actually charged by the hour.
4:46So we had no local free calls as in other places. So it was a very big deal to get a cable modem. And when I got it, it was very much delayed and the cable company had issues with it. And when they had it going, I started scanning their infrastructure. And I found out that while they had finished setting up their internet, they had not finished setting up their security. So I could sort of waltz right into their servers and stuff. And so I sent a message to the system administrators using the famous wall command. I said, hello, I'm here. And they said, well, if you're so good with security, then come over.
5:20And so I got on my bicycle, this being the Netherlands. So I drove to the cable company. And on my way there, I was like, yeah, this could end up really well. or I could end up being arrested. So I just waltzed into their systems and it ended up really well. A little bit too well because that was the dot com boom times when there was such a shortage of staff that I came in, they gave me a chair and he said, sit down. And that actually ruined my studies of physics because then I had this job over there. But it was quite cool. It was very nice to scale an internet service provider from like 50 users to 50 ,000 users.
5:55I mean, everyone should maybe one day scale a company by three orders of magnitude. It's very educational. So I had a great time there. And it was a wonderful way of sort of learning how to run an internet service provider. Yeah, from scratch, which is very educational. Because if you mess it up at that stage, it's not so bad. But if you mess it up by the time, like Comcast scale, it is pretty bad. You can't mess around at that stage. You got to have the uptime, right? All the nines. So what were some of your learnings as you scaled up? These are kind of like the first time things are scaling up.
6:32So you can't exactly go read a book about it, right? Yeah, I actually wondered if anyone had written the book about it. But I don't think we've had that rate of scaling that we had in 2001, where you would double every month or something like that. One of the main things I learned there is that it was quite amateur stuff. So you need to professionalize and figure it out. So for example, at one point, I did take down the whole internet service provider because I had fixed a bug in the LDAP server and I was very pleased with myself that I had fixed that bug. So I shut down my laptop and I went home.
7:07And I had forgotten that I left the LDAP server running in the foreground. And the moment I shut down my laptop, the whole internet service provider shut down. And on my bicycle back home again, I was barely out of the building before they called me and said, what did you do? And so I had to rush back in. And that was, I think I, well, I learned a lot there in the sense that you really need to, it's nice to hack yourself around these bugs and then feel really good. But it's also maybe really good that once you fix the thing, then to also sit down and really tighten things up again and not just shut down your laptop and shut down the whole internet service provider.
7:46Well, unintended consequences, right? You just don't know what's going to happen. and shut that laptop think you're done for the day one really interesting thing that i learned there and i've been passing on this message since forever and i'm also going to pass it on to your listeners okay so we had a redundant email setup and we were yeah we thought it was good it was redundant so there was this guy in charge there and and he said well if are you sure this is redundant i said yeah it's all redundant and then he pulled the plug of one of our servers and then everything broke because it turned out that it was sort of yeah we felt that it was redundant but we had not actually had someone pull the plug on us and see if it actually was redundant and then it turned out that it was not redundant and since then every time someone says hey we built this redundant system i say can i just go in there and start shutting down some random servers now is that is it gonna fly and they say no no no no don't actually be doing that I'm like, well, maybe you have to work a little bit more on your redundancy until you can live with me going in there and just shutting down some random stuff to see what happens.
8:48So that has always stayed with me. If someone says it's redundant, I'm going to ask you, can I just shut down some stuff for you then? Yeah, that's not a good thing. You know, pull the plug. Things fall over. Yeah, yeah. Well, it's a good thing. I mean, if people say, look, it's sort of redundant in the sense that we can make it work again. Okay, that's fine. That's fine. That's my level of expectation then that something could fail over and you could rapidly make it good again. But if it's not like, but if you're telling me that, look, this is going to be automatically redundant and it's going to work anyhow, then I should feel free to just unplug some stuff and see what happens.
9:24We all learn those lessons to some degree, right? Like whenever you think something is safe and secure or redundant or whatever you might want to call it. And then chaos engineering happens essentially, you know? Yeah. Yeah, yeah. And you only learn that. You only learn that. So recently I've been, I mean, we might come to that recently. I'm now running some actual services again that people depend on 24-7. And everyone has to sort of live through that for a bit. So like experience being on the line. Like, you know, yeah, people are relying on this stuff and I should be doing a good job here. And that's a whole new level of thinking that you get, which is very different from a programmer that just programs and never get those 3 a.m.
10:06phone calls. This is why I've been trying to convince Adam to run our Zulip community from his home office. You know, like just host it for us. Self-host that sucker. We only got a few hundred people using it, but they're using it all throughout the day. For the learnings? Yeah, and for that upgrade of yourself, like just what it feels like to like have people depending on your service. It just changes you. I'm not against it necessarily. It's just I prefer not to. I mean, I get that feeling. I really understand because it's it really sucks because once you have that stuff running in your house you have to start planning around things because you cannot just have an electrician come over and do some redo your wiring like yeah we're going to be down for three hours I mean it does make me think so straight out of university I ran some mail servers on a Linux network for a company maybe a few hundred people and there were two servers and they worked in conjunction there was spam assassin and it's all kinds of stuff.
11:01It was post fix. And when we had issues, it would just queue up, you know, like as long as the servers were online, even if a mail server is offline, it comes back on, you know, that delivering server, they're going to try again a little bit later. And so it would just queue up. And so there were times where I would come into work and I would look at the mail queues and people are thinking it's a quiet morning. I don't have any, I don't have any email this morning. It's like, Oh, you do. I just haven't been putting them in your inbox and then i just i just love just watching that queue just work its way down back to zero but definitely know the feeling of like other people relying on your on your network services it it can be stressful it can be yeah unnecessarily if it's unnecessary you know unnecessary right if you have someone else to do it for you why do it right bert what are you running these days you said you're running some stuff now yeah so so i built a sort of parliamentary monitoring system so we we have a also an upper chamber and a lower chamber and politics house here and they have a pretty competent website so it's not that it's bad but uh like half a year ago at one point there was a new law here in the netherlands which is quite impactful for the internet community and we hadn't noticed so the law was about to enter into force and we didn't know about it and luckily the law was not so bad but we only had a few days to look at it and we knew that if it was bad we would not have been able to do anything in those few days so back then i thought like and i we can never have this happen again and on the dutch parliamentary website you can put some some search terms and it will send you messages if any new documents match those search terms but it's a bit clunky it's not that great so i thought let's make some kind of advanced warning monitor that we on the internet it can put up just searches and say like the moment you have a meeting about internet censorship i want to know about it and not only do i want to know when when it actually happens i want to know as many months in advance as possible and from that i built this this monitoring website and it grew and grew and grew and actually the sort of fun thing is if you are outside the government you can innovate rather quickly yeah and the dutch parliament is completely open data so everything they do uh you can you they you can effectively monitor the state of the parliamentary documentation system to your house, which I do.
13:27And that grew and grew and is now quite popular in the sense that even many politicians are using it and many civil organizations are using it. And the most interesting thing and which I find most important, it turns out that my simple parliamentary information website, and it's like really straightforward, it's html almost no javascript um works really well with software for blind people and it turns out there are some government workers that have to work with these parliamentary documents and they found out that their screen reader software just has a lot of fun with my site easier to read on your site than the official site so i now have a few uh low vision government workers that are actually relying on my site to do their job and that has added to my sense of that is not just a hobby anymore it's just that people actually need this stuff and um so yeah this is and there from that i learned all these very interesting things that you only learn when you go live with services with things that you don't know for one thing that for example that i learned is that if you have an iphone and someone enters a search term in your form and you do that within quotes because you want to search for for a sentence and the iphone will turn it into smart quotes.
14:46And that is not something your search engine is ready for. Your search engine doesn't recognize these smart quotes as smart quotes. Well, it just puts them in the smart. So I had this weird situation that iPhone users couldn't find anything. And this is the kind of thing that you only learn in production. And the other thing I learned, I have these signup emails where you can and passwordless login emails. So you can just log in with your email address, which is quite popular these days. And for some people it didn't work. And it turned out that Microsoft now runs a security scanner that will actually attempt to log in for you.
15:24We know this very well. We were just talking about this topic on the previous episode with somebody, but you have new information now because we have the same thing, passwordless logins on our website. And what my trick to fix that was, was I switch the actual request from a get to a post and you get, but I read on your website that that's not going to even work anymore because Microsoft is now executing JavaScript in their... They're posting to your website. They're now posting. And they're posting. And the weird thing is, so the strange thing is they do the post and which is already, I think, violating many people's assumptions.
16:02Yeah. You should not be posting on behalf of anyone else. But the other thing is when they do that post, my site actually used to return a cookie, a session cookie, which means that Microsoft with this security measure. So the reason they do this is they want to see, is there malware on this site? And might that malware only pop up after a post? Okay, well, I see where they're coming from. But when they send that post to you, my site would used to respond with a session cookie. They said, well, welcome, you're logged in now. Which means that Microsoft is receiving tons and tons of these session cookies right now.
16:40Cookie monsters. Yeah, but you could actually do. These cookies are very valuable because these are the session cookies that allow you to do stuff. Well, it now appears that the new barrier is they will execute your JavaScript. Okay. They will execute your posts. Okay. Okay. But they will not, for now, click on a button. so you must have a button in there right now that and that button then does the post okay so you are actually you're adding a step for the real people because now they have to click on the link in their email and then come to your site and then click on a button to sign in yeah and but but but there's no no one no microsoft did not announce that they would be doing this and they have also not announced that they're not going to click on buttons so maybe one day they will click on buttons Well, you need a monitoring website where you can monitor Microsoft's changes.
17:29Yeah, well, it is. But it is the kind of thing, and I cannot stress this enough, this is the kind of thing that you learn when, as a programmer, you go into production again. 100%. And I've since heard many people, they told me that Trend Micro also does this. And actually, I ordered some hardware stuff from my store today, and they have a link that is vulnerable to this. and uh and when you have to click that says i'm going to collect my uh my hardware now and that is already useless for them because microsoft is doing all the clicking right now because microsoft's already registering everyone's hardware for them yeah yeah yeah it's a crying shame it's tough out there in the real world of the internet isn't it i'm going back to your monitoring site what formats does the government put stuff out are you like diffing html or is it better than that oh this is a story this is so on the one hand they have a glorious api and and and and actually i didn't read the manual they use this thing called open sync or something like that and that is apparently a sort of weakly determined standards by which you can replicate a relational database to somewhere else as a series of xml changes and and you can pull these you can say I want to get all your changes since marker, such and such.
18:47And that's actually pretty nice. So it is quite convoluted because I think it would have been easier if they just said, look, this is our SQL database and you can query it. But now you get this stream of XML messages and that is actually quite glorious and good. Now is where the problem comes. Governments, when they send you documents, they love PDF. PDF has a lot going for them. You have PDF slash A, which is an archival form of PDF, which is actually nice because that's the sort of PDF variant where you can say, I'm sure I will be able to read this PDF 50 years from now. So it only uses built-in fonts.
19:26It has no language. It doesn't do anything. So governments really love their PDF files. The thing is, web users do not love their PDF files. Web users want to see web pages. They want to see HTML. They just want to click. And the thing is, when the Dutch government gets a document, it starts life as a Word document, so DocX, which is easily processed. Then they convert it to XML, which is also easily processed. And then finally, they convert it to PDF, which is not so nice to process. So in order to make all this stuff work, I found out that there is a sort of official government archive of government documents.
20:07And there they also publish the XML. So in the course of this trajectory, I retrieved the document like four times. And to save it, I have to retrieve it from some kind of official government archival site where there is XML. And that XML actually is glorious. So when someone speaks in Dutch parliament, they make a note at the exact timestamps when they speak. And this is so they can match it up with the video. But it also allowed me to make these statistics and say, who are our sort of Congress people that talk the most or that talk least or that have the longest sentences? And because they log it all so well, I could also, who are the fastest talkers of our Congress and who are the slowest talkers?
20:52And this is, of course, not very necessary, but it is a lot of fun. But yeah, I never used to be a fan of XML, but it actually does actually get the job done in this case. It's actually not so bad. Yeah, relative to PDF, it's better, right? Yeah, well, everything is better than the PDF. I mean, people, I understand that people sort of love it because it feels like it's a standard format and you have everything in there that you need, but to process it, it's nasty. I mean, for example, if you have a two-column PDF file, it's actually not easy to figure out that it is a two column pdf file because if you look at the postscript it's just a sentence and a bunch of spaces and then another sentence which is the next column so it's all not that great so do you all have the filibuster over there actually we no we don't and i i try telling people that we our government is our politics are also becoming wilder and and yeah less useful but we we haven't yet sunk to the level uh that we need a filibuster kind of thing so for now if you have a simple majority you can actually get stuff done in dutch government but i worry where it's going i always thought the filibuster was just the dumbest thing it's like if we can just talk long enough we can run the clock out it's so yeah silly but i ask that because then do you have any blowhards like do you have a ranking of like Like, here's our most wordy politicians versus not.
22:16Yeah, yeah, we do. And we have these people that actually, well, they will talk your ears off. And actually, it's not always who you think it is. It is sort of funny to, when you run these numbers, you find out that it's actually quite often that it's different from what you thought it would be. We all need in government is a haka. Oh, excuse me? A haka. What's this? I may or may not be pronouncing that properly. Have y 'all seen this in New Zealand? This last, like, tail end of last year? Yeah, no, that was very impressive, and they should do more of that. Oh, my gosh. I'm not sure the context of the protest, but very much a spectacle, and I think we all need versions of that in ours.
23:01Describe it, Bert. Describe it in your best words. Well, they had this in New Zealand. That's what you're referring to, probably. Yes. Yeah, so they had a law that was passed, which was not good for the indigenous people in New Zealand. And then they got really upset. And then the indigenous people or the people affiliated with them in parliament actually performed a whole song, a protest song in government, which is really a very important thing in New Zealand. But it was, I think, a great way of making a point. And I have not, I cannot imagine US Congress breaking out into a song. Oh, goodness gracious.
23:37Please, no. I don't think, I'm not, this may be song. but I would not describe it necessarily as song. It doesn't sound very nice. It's more like a screaming song and like a dance and performance. Like a chant, yeah. It's probably more akin to a chant. Right. Very performative. One person began it.
24:15Great demonstration. I think it's turned into a meme in terms of how to protest. She tears up whatever it is they're protesting. And then many people that are in the parliament space begin to join in with this dance that turns into song and dance. Wow. And so it's very, it demonstrates very well. So if you feel very strongly against something, clearly it's got a very visual and a visceral process to like make you pay attention, you know, so it's interesting. So you should check it out, Jared. I will. I'll go check that out.
25:03well friends you can now build invincible application thanks to temporal today's sponsor you can manage failures network outages flaky endpoints long-running processes and so much more ensuring your workflows and your applications never fail. Temporal allows you to build business logic, not plumbing. They deliver durable execution and abstracts away the complexity of building scalable distributed systems and lets you focus on what matters, delivering reliable systems that are faster. An example of this is Masari. They are the Bloomberg for crypto. They provide market intelligence products to help investors navigate digital assets.
25:39and they recently turned to Temporal to help them improve the reliability of their data ingestion pipeline. This pipeline collects massive amounts of data from various sources, and then they enrich it with AI. This process previously relied heavily on cron jobs and background jobs and queues, and the design worked well. However, these jobs were difficult to debug at scale because they needed more controls and more observability. And as they looked to rethink this ingestion flow, they wanted to avoid cron jobs, background jobs, queues. They didn't want to create a custom orchestration system to oversee and to ensure these jobs and work was being done reliably.
26:19Here's a quote. Before Temporal, we had to code for dead letter queues, circuit breakers, etc. to ensure we were resilient to potential system failures. Now we eliminate these complexities. The headache of maintaining custom retry logic has vanished by using Temporal. End quote. So if you're ready to build invincible applications and you're ready to learn why companies like Netflix, DoorDash and Stripe Trust Temporal as their secure and scalable way to build and innovate, go to Temporal.io. Once again, Temporal.io. You can try their cloud for free or get started with open source. Once again, Temporal.io.
27:01Well, one thing about governments, it seems, as they go on, they either incline or decline, and many of them decline. And this is a very poor way of segueing into the topic of long-term software development, which is something you've been interested in, Bert, and I'm also interested in, as we have systems that exist for decades and become quite crucial. Software has never been more crucial in the life of everybody out there in the world than it is today. And a lot of these systems are not built to last or they weren't built to last or they thought they were building it to last, but they actually weren't.
27:38And you've done a lot of thinking and even writing and talks about long term software development. And I think we would all benefit from hearing your learnings and what you've been talking to other folks who've been experienced or believe about this. We all have thoughts about how you actually write for the long term. I'm curious what you have found. Yeah. So it was invited. So I'm a very part-time technical advisor to the Dutch government also, to what you would call the FEC, the electoral board. And they built this software for tabulating the national elections. And they are revamping it. And they said, look, would you take a look at our code?
28:20Can you tell us what you think about it? And I looked into it and I found that they were doing sort of more or less standard 2025 software development. And then I wondered, and that means having like 1600 dependencies. Because if you have a Node.js built project these days and you install it, it will, for starters, pull in like a thousand dependencies. And then you haven't done anything yet even. And so I looked into that and I thought, well, this is not going to be something that's viable for the next 10 years. because these dependencies change all the time and NPM might go away for the next 10 years.
28:58So I turned that into a study. What would you do these days if you had a 10-year software project? And I went to Mastodon, that is really a good social network to ask these kinds of questions because they're full of nerds and they're my people. And so I asked her, I said, look, what are your thoughts for anyone doing long-term software development? And we had people way in that have a software with an 80 and not 18, 80 year horizon. Wow. And these are people that study and have to measure the radioactive decay of nuclear waste. And they are tasked with measuring and supporting that for the next 80 years.
29:41And so they spent quite some time thinking about this stuff. And it turns out that actually it's sort of modern software development. It's like sort of the complete opposite of what you would do if you want to have software that works 10 years from now. And in the sense that we have these huge, that even building software is fragile right now. So if you just, not even that it does the same thing, but if you want to build a sort of standard 2025 software project, you need a working internet connection and like hundreds of servers around the world that supply you with data even before your software runs.
Read the full transcript
30:16and that is not something that is a healthy thing to have if you want to promise people that it will work 10 years from now. So I got a bunch of very strident feedback from people. They said, well, if you look at the problems that people have with old software, something goes wrong with the software. There is a problem. Something always goes wrong, of course. And then in your mature software project, you need to figure out what is actually going wrong. and the most what almost everyone said is keep it simpler that was of course we all know that we should not write software that is as complicated as we can make it because that's not going to end well but everyone said look we are like had very bad experience like we need to figure out seven years ago why this clever code what it actually does and then often you find that this clever code, there was no need to make it clever.
31:12So if you have like 10 political parties or in the Netherlands, maybe fun, we have like 25 active political parties. So that's, I mean, you have two. Yeah, but can we borrow some? Yeah, well, yeah, you could. Well, having 25 is not an optimum. I can tell you that. But even if you have 25, there is no need to set up a complicated data structure to hold 25 political affiliation names. But one day you might sit there and say, hey, wouldn't it be useful? Wouldn't it be fun if we had a complicated red-black tree so that we could make really rapid searches of our 25 political parties? And then you sit there and maybe in 2032 trying to debug why it doesn't work.
31:53And that's because you try to be clever. And like so many people came to me, they said, if I have one regret, it's that we built things too complicated. And then seven years later, no one understands the code anymore. And the other thing that people came up with is that people change jobs a lot for a number of reasons. But if you want to do long-term software development, the easiest way to do that is with long-term employees. Because they build up knowledge, they know how the stuff works. And of course, but if you have people that are changing in and out all the time, then you become very reliant on good documentation.
32:29And it's, of course, always good to have good documentation. By me, all means do it. But it's very hard to compensate for all that knowledge walking out of the door after two years. So they also, people said, look, you really need to, I mean, it's old fashioned and it's almost crazy talk these days, but you really need to just invest in keeping your developers happy and wanting them to stay for like maybe seven years or eight years or something like that. But again, this is also not how many places work these days anymore because they have contractors and they zoom in and they zoom out. And it's very, so it's actually, so as a result, so I wrote the article based on my presentation to the Dutch electoral board and it hit Hacker News and other people.
33:19And amazingly, almost everyone agreed with it. It was also very rare that it happens. but like 30 ,000 or 50 ,000 people read it. And based on that, I got a lot more feedback from people that said, look, we are doing nothing like this here. We just, we are following sort of the Facebook model of software development where we deploy all the time and most days of the week it actually works. And sometimes we go through a bad patch. But then there are all these kinds of people that have to make software that opens and closes bridges and doors and MRI scanners that could physically tear themselves apart or pacemakers or robots that actually do surgery.
34:04These people are like, no one understands us anymore. Because if we do the regular software development thing, they start with stuff that is so fragile that we would never entrust that to open or close a bridge or launch a rocket or whatever. So it's actually becoming quite a rare scale. to build the kind of software that is has like safety of life consequences to a bunch of people that that will start with downloading the latest javascript framework even though that might be very nice for your website a lot of great stuff in here uh dependencies you know choosing those wisely keeping it simple obviously these are all i suppose like tried and true kind of to some degree well known so it's not like you've you know reinvented the the simplistic wheel you just sort of collected what might be a good zeitgeist for folks to follow yeah i collected very it is a list of extremely obvious things but once sufficient people have sort of forgotten about the obvious things it starts becoming worth your while to just restate the obvious things and say look you're not you're not crazy you're not i mean because many of these things are are obvious but also sort of old-fashioned but it's i think i i think it was worth restating also it has given it has strengthened the spine of a number of people um that that that it has reinforced them they say look let's let's work on keeping it simple uh let's refactor this stuff and let us not optimize these things before they need to be optimized right that's a big one it's easy to get nerd sniped into premature optimizations.
35:44If we just focus on the simplicity part, let's return to dependencies and maybe talk about that in detail. But I don't think very many people set out to make a complicated solution. Like it's not, unless you're just malicious or like willfully ignorant or something, like you're not like, I'm going to make the worst possible solution ever. We all want to make it simple and straightforward and repeatable and reusable. And like, these are all things that engineers, software developers strive for. And yet it seems like we all end up making things that are too complicated anyways, right? Like nobody's like, I'm going to make a bunch of spaghetti code and it's going to be a terrible mess four years from now.
36:24And yet we end up there. And so do you have, did people come up with ways that you could actually like deploy a strategy or a tactic towards simplicity? Yeah. So I would not actually agree that we all say we like simplicity. This is true. But there are, for example, people that take a language standard and see it as a personal challenge to use all of it. Because maybe that demonstrates proficiency. Yeah. Yeah. It's like, look, I'm really good at this stuff. And see how complicated this code is. And I think the only people that do this correctly are the people that feel it in their bones. that when they're typing in something complicated, they worry already about the 3 a.m.
37:12debugging session that's going to ensue five years from now because they've been through it. They've been burnt before. Yeah. And so I think there are actually a bunch of people that really, maybe it's controversial, but for example, programming in Rust is all the hype these days and it has many things going for it. But when I tell people, say, look, it's quite a complicated programming language, they actually like it that way. it's a challenge to write in there. And some people actually sort of enjoy writing challenging code. And so I think the best way of writing simple code is having been on call.
37:52Right. That is like the one way of keeping yourself pure because you know, if I mess this up, if I write this too complicated, it's going to cost me my sleep one day. Basically deal with the mess. You know, you do that in product. You say, hey, do support for a bit. Hey, if you're an engineer, be on call. You know, deal with the mess that you create. Because it's pretty easy to like write something, throw over the wall and walk away because, you know, you're not on call. And in many organizations, there is actually a physical wall. It's just the people that write the code will never, they will only indirectly hear about it if there are any problems with it.
38:29And that means that there's a natural tendency to keep it simple because you know you're going to be stuck with the mess. If you put an organizational barrier between dealing with the mess and writing the mess, then it maybe becomes a lot more attractive to try this new style of programming that is like impenetrable. And that goes back to your churn mention as well. Like if you're only going to work somewhere for 12 to 18 months, unlikely that you're even going to deal with the mess that you're creating because you moved on to the next organization by the time the mess rears its ugly head. And so you actually may be thinking that you didn't make a mess because you're gone.
39:06Yeah, you never had the opportunity to find out. You may have even gotten a bonus because you delivered in such a time frame. Very complicated. Yeah, it's so hard. Solutions out there solving the problem. But meanwhile, there's something percolating behind the scenes. So ego kind of gets in the way. Like we want to be clever. Yeah. And sometimes I write codes that is so embarrassingly simple. And I'm like, it cannot possibly be enough. And then like five years later, you go like, well, apparently it was simple enough. Apparently it was good enough. And there's also this sort of anticipatory thing.
39:41So for my parliamentary monitoring system, I have a very interesting situation that in parliament are like seven or eight different kind of important entities. There are meetings, there are documents, there are appointments. and seven or eight is that the tricky thing about that is can you write a useful abstraction that is useful even when you only have seven sort of different things and you could write this lovely generic solution where where you could have thousands of different kinds of entities in there and but it would be quite complicated and but because there are only seven or eight entities, it is not always necessary to build this sort of generic solution, but it does feel a little bit bad if you have a switch statement with eight kinds of categories in there.
40:33It feels a little bit like, no, I should have built an object and inherit from that and made it all generic. And that's also the one thing I see that people are sort of anticipating huge complexity later on, but that might never happen. You might never get to that point where you need a generic customer service class handler factory. Right, yeah, Agni just rears its ugly head. So I've had this, have you had this before, Barrett, where it's like, I want to create a simple solution. I go ahead and do that. I put it out in the world and it's too simple because the world's actually more complicated than my software will handle.
41:13And then as it kind of grows in order to handle these different edge or corner cases that I didn't consider when I was building Simply, it ends up all hairy at the end because it's like, well, it has to deal with 17 different things that might happen. And it's like, well, was there ever a simple thing that would have solved this? Because the real world is complicated sometimes. And then I throw up my hands and think, I don't know. How am I actually going to do this? So there's one sort of telling story. So I also wrote PowerDNS, which is a name server. and we started PowerDNS in 1999. And DNS evolved over time and it added cryptography to it in DNSSEC.
41:54And initially PowerDNS started out like really simple and then the world turned out to be complicated and we added more and more complexity to support this cryptography in there. And it was very difficult because DNS itself, for example, is highly case insensitive, case really doesn't matter until you start doing hashing and encryption because that's sort of, yeah, it cares about case. And usually you sort of end up then, and as you described, you end up with a situation that it's just a mess. And it's very tricky as an organization to allow yourself to say, no, we're going to clean up the mess.
42:35So we're not going to do this big, big hairy, we're not going to do version two syndrome where we say, no, we're going to make the best version ever that will be so generic that it deals with everything because it doesn't work. But what does work is actually sitting still and say, look, we're going from version two of the software to version three, and we're going to give ourselves a whole year to just do spring cleaning, house cleaning. And we're going to make the things that were too complicated, we're going to make them simple again. And back when we used, for example, in PowerDNS, we used simple strings to deal with domain names.
43:09It turns out domain names only look like strings. They actually have more complicated semantics than that. So within PowerDNS, the big thing we did was we said, look, we're going to rip out all the strings. And we're going to move to a dedicated DNS name class that's going to deal with all these weird DNS things. And the only way this could happen is by allowing ourselves like a year to do a cleanup. I can very much recommend doing that. Did this turn into a business? Like was this making money at the time? Yes, it turned into a good business. And actually PowerDNS is a rare example of a fully open source piece of software that actually turned into a commercial success while still remaining mostly open source.
43:54So were there pressures on the money side to like not do that? Yeah, there were. Yeah. But you have to sort of have a little holistic view on that because it turns out that we could have focused on where we were making all the money. But it turns out that the reason we got paying customers was that the technical community liked what we were doing. They would recommend us everywhere. So if we had focused on purely the technical, purely the financial bits, we would probably have had great business for a few more years. But eventually, we would not have been able to do proper DNSSEC. No one in business cares about DNSSEC.
44:32They really do not care about that. However, the nerds, the technical people that work there, they really do care about that. And we also cared about that. So it was, in the end, a solid business decision for us to say, we're going to give ourselves a year to clean this stuff up. And I dare say that if we had not done that, the company would have gone bust later on. It would have collapsed under its own complexity. but i've i do not find a lot of other examples where people gave themselves like a year put the old software in maintenance mode and we're going to just do cleanup for a solid year and then come back better that is actually a very rare thing that's rare the only one i could think of is snow leopard which was apple's version of mac os that was touted as having no new features even though there were a few features in there but it was an entire release cycle all focused on stability refactoring performance oh well and it is to this day many people's favorite version of mac os going back like everyone's like yeah snow leopard was awesome yeah yeah well that i think as a concept is doing a spring cleaning uh yeah we do it for other things i mean in on equipment you do this kind of maintenance uh where you replace the the load bearing parts and and and take it out of service for a bit so yeah but but it's not very popular i mean mostly you see you see people just having the version two second system syndrome where they say look now the next version is going to be perfect they're going to redo everything from scratch and then they forgot what made the old version great that's also a thing but just doing incremental cleanup yeah i very much recommend it reminds me something mechanics say that i learned recently they say if you don't schedule your maintenance then your car will schedule it for you yeah exactly that's actually also part of the blog post i wrote on the software oh is that where i read it okay that's probably where you read it the thing i put it in there i got it from an australian guy and he said if you have dependencies in your software schedule some time maybe every quarter just to take a look at all your dependencies how are they doing are they still being maintained are the security releases coming out um has the new vc started the monetization cycle yet and and the idea there was that look if you do this on your time schedule you find out ahead of time that things are not going well and the other way is that you figure it out because the stuff doesn't compile anymore yeah right when i was in the military we had this really awesome term that I still live by.
47:10It is PMCS. It stands for preventative maintenance checks and services. And legitimately, like every time I got into a vehicle, I could not just get into it and just drive. I had to do this series of checklists every single time because, you know, if you don't, eventually you get bitten by this non-preventative maintenance checks and services. I had this thing called a Humvee. It was a very big vehicle. Yeah, I was in, And I carted around a lot of diesel fuel. In the military, we called it JP8, but it was diesel basically. It was 2 ,500 gallons of very expensive fuel that I had to transport safely, securely.
47:49And if my vehicle wasn't capable of getting to and from, or there was something wrong with the tank or just anything, catastrophe could have been right around the corner. So we had this PMCS mindset just ingrained in us from emptying out connexes to just confirm what was in there last week is still in there. Even though everyone knew it never got opened and it never got used, we had to pull it all out, confirm it's there, and put it back. It's just like the boring stuff. You have to do that mundane checks, boring stuff to make sure that what's on the rails stays on the rails. But interestingly, it's a very interesting, good concept.
48:28I sometimes get that reaction when I tell people about unit tests and regression tests. They say, well, look, the software worked the last time I used it. So it's still going to be working. And I'm like, well, just write some unit tests and see what happens, which is basically the sort of same thing where you just ask the computer to add two and two and you check if it's still four. But still, so often unit tests will find that you broke stuff, even though you thought it impossible that something got broken. And so I do enjoy that mindset where I say, look, we're still going to test it every time and do that.
49:02And you never know what happens behind your back. But there are people that are like, you know, I'm not going to write unit tests because my software worked the last time. Right. That test suite becomes such an asset down the road. I was just updating our dependencies today and just having test coverage where I could say, OK, here's a dependency. run the tests, everything passed, update it to the latest, run the tests again, everything passed. It just has a peace of mind that you're like, okay, updating that dependency did not break everything. No, and it's magical. PowerDNS was, of course, very old.
49:36And in 2001, we made the first regression tests and unit tests. And that was like an amazing feeling that your tests discovered that you made a mistake and they told you before you shipped. I mean, it's still a good feeling. And I sometimes give training to scientists on programming. And when I show them the power of unit tests, they go like, wow, that's just awesome that you get an email message that you made a mistake instead of getting a complaint later on. Yeah, it's like if you don't run your unit tests, then your customers will for you just to reapply the statement. Yeah, that's the same kind of concept, yeah.
50:33Well, friends, I have a question for you. How much of your personal information, your private data, how much of that is out there on the Internet right now for anyone to see? I think it's more than you think. Your name, your contact info, maybe even your social security number, your home address, potentially even information about your family. It's all being compiled by data brokers and it's being sold online. And these data brokers, they make a profit off your data. They sell it as a commodity. They don't care. Anyone on the web can buy your private details. And this can lead to identity theft, phishing attempts, harassment, unwanted spam calls.
51:08I get those so much. But now you're able to protect your privacy with Delete Me. That's today's sponsor. I recently found Delete Me and they sponsor the podcast and they offer a subscription service that removes your personal info from hundreds of data sources online. Here's how it works. You sign up and you provide Delete.me with exactly the information you want deleted, and their experts go and take it from there. They send you regular personalized privacy reports showing you what information they found, where they found it, and what they've removed. And it's not just a one-time service. Delete.me is always working for you, constantly monitoring and removing the personal information that you don't want on the internet.
51:47To put it simply, Delete Me does all the hard work of wiping your data, your family's data from data broker websites. So take control of your data and keep your private life private by signing up for Delete Me today. Now at a special discount for our listeners. Today you get 20 % off your Delete Me plan by texting CHANGELOG to 64000. Once again, text CHANGELOG to 64000. zero zero and as you may know message and data rates may apply see terms for details enjoy
52:25here's a question for you bert how do you know how many dependencies is too many dependencies like is there a heuristic where do i know that i've just jumped the shark i i struggle with this myself um because i i do a lot of reinventing the wheel um often because i know that i could spend the time learning about your dependency and then i would still be disappointed because like two years from now the dependency would be different different again so i i do often spend more time than is wise probably re-implementing some simple things i draw a very hard line at cryptography, for example. I mean, there is, unless you are like a famous cryptographer, you should not be doing cryptography.
53:10You should leave that to someone else. But even then, if you look at the state of many cryptographic libraries, OpenSSL has been a sort of disaster zone for years now. But I draw a very strong line there that you do not build your own cryptography. But on the other hand, if someone asks me to be responsible for a piece of software, and it's important piece of software. It's not a birthday calendar. It's something that people rely on. I need to know what is in there. I need to know what I'm shipping. And if I run a modern piece of software development and I end up with like 300 dependencies, which are also included dynamically.
53:50So at one point, I was in a situation where we said, we have to audit this software. I said, okay. And then it turned out that if you would build the software three weeks apart and you didn't change anything, it still would have changed because by that time, the dependencies had moved around and shifted their own dependencies. So I'm actually sort of hardcore in the sense that I don't think you can ship software with like 300 dynamic dependencies. And people are still doing it, of course, but I do wonder what they're doing. And people have said, look, you C++ people or C people or sort of, you have one huge dependency, which is called your library.
54:37And you rely on your operating system for a lot. And that's also like millions of lines of code. But there you have a little bit of safety blanket that everyone relies on the C library. And so there are a lot of eyes on it. And it also doesn't change that much. And we all rely on the operating system. And lots of people are looking at that. So I'm sort of maybe on the low end of rebuilding my own wheels too often. But on the other hand, I still cannot understand shipping software where you say, look, I honestly do not know what is in there because it gets determined at build time what is in there.
55:23And maybe you could audit 300 dependencies if you would have enough. But it requires a lot of work to actually sort of even figure out who is writing all these dependencies, who owns them. So, yeah, I struggle with this because when I tell people that I simply cannot believe that you run NPM install get, I counted it, five million lines of code come out when you do that. for the most basic product. How can you ever know what is in those 5 million lines? But apparently I'm old, apparently, and it is now considered very normal. And I know for a fact that my car has NPM in there, and I find that worrying.
56:12How do you know? Because it tells you so. There is this screen that says the copyright and the licenses and stuff, and it says, look, React is in there and NPM is in there. And I hope it's not in the driving parts of the car. I hope this is just a radio or something like that. The entertainment sections of it, yeah. I hope that is the case. I hope that is the case. Yeah, the tough thing about dependencies, even when you can audit, if you can and have audited and checked out the owners and the code and the whole do your due diligence, is that what we're learning over the course of years and decades is even if you do that, if you're loading your dependencies from the network, you can't trust the network.
56:57And you can trust what you think you can trust today, you can't actually trust a year from now because the network changes. And so even if you're doing some due diligence, like you can still get bit, these so-called supply chain attacks are happening more and more often where all of a sudden what you thought was your dependency is replaced with code that is not the same code. And that's incredibly troubling for me. Yeah, it is. And I come from a world where people have to also have to develop on networks with no internet connection. And it's just basically impossible for many people to function there anymore because you try to do something and yeah, there is no internet.
57:32There is no network. And the people that are in national security software or encryption software or that kind of stuff, intelligence agencies, they are actually in trouble because the art of developing without a fully functioning network connection as dying out because there are lots of people that they need a runtime connected to the internet to get anything going. And even then, your software might be relying on third-party services, CDNs somewhere. And if you're running this nuclear power plant where people say, look, we're not going to connect you to the internet, it's becoming a vanishingly rare skill of being able to build software that runs without a network connection.
58:18One thing I appreciate about this part of your post on dependencies was just the ways that you enumerated over what might happen over time with dependencies. I'm going to read them if you don't mind. You said just basically, you know, scrutinize and limit your dependencies because all of them might over time. And here's a list. Drift away, leading to adjustments in your code or worse, silent changes within behavior. Jared, you kind of mentioned this. shift to a new major version with cemented changes requires rewrites on your part it might get abandoned or simply disappear or start to decay it might get hijacked by a nation state we've seen this happen in recent past by nation state actors think NPM PiPi etc.
59:04it might start to get monetized by the new VC owner who's I guess acquired the open source or the dependency in some way shape or form and it might develop conflicting dependency requirements of their own. I think just enumerating those different things because you think, okay, obviously, the simple answer is just simple software, linear dependencies. But why? And that's the why of those. And it's not all the whys, but it's several versions of the why that might help you understand why you shouldn't. Yeah. Yeah. And these are, I mean, these things all happen. So I have, I've worked with an impossible Python project that just requires two different versions of the same dependency.
59:43and they cannot do it. So there's now, and you also don't know how to recover from that. And, but on the other hand, it is of course sometimes great fun that someone just moves ahead so quickly because they embed all of Chrome in their project. Yeah, boom. And then I'm sort of this ascetic person that then needs to be really, I need to work really hard to display something that looks like a webpage. And someone else has no qualms at all about, embedding an entire electron copy. So it is always a bit of tension. It says, look, I'm telling you about these things that will be bad like five years from now.
1:00:24And these people, hey, but I'm shipping today. So I do really feel that stuff. But especially lately we had InfluxDB, I think, which all of a sudden, which is a time series database. And I think that they have now limited you to 500 metrics or so. because they want to make money and of course i respect that they want to make money but if you build your whole project on influx db uh yeah you have an issue right now because your business model might not support their business model is this a recent change that happened or is it something that's yeah it is i think it's launching from from the past two weeks or something like that right and um which is why i tend to do almost everything with sqlite uh because we know sqlite is going to be around forever.
1:01:13And we know the people that run it and they're not going to do anything like that. They don't want to do anything like that. So there you know that like I can, I mean, a hundred years from now, people will be able to read SQLite files. It's that abundant right now. So then I like, I try to stick with these things that way you be sure that look, 25 years from now, they will have this software around. And so it's not going to bite me. It kind of leans into some of the things you mentioned too down the line, which is more of a decision tree of sorts, which is how to scrutinize these dependencies.
1:01:49You know, what does the tip look like? Who else uses it? Who writes it? What are their goals? I'm kind of wondering if there's something out there that a piece of software we can sort of throw a dependency at it and it spits back, you know, some versions of all these answers, you know. Actually, that would be good. So we, of course, have these sort of S-bombs right now. the software bill of materials, but that is only like, yeah, it's this piece of software and it's that version. But it would be indeed be very interesting to say, look, this software is actually funded by this foundation or something like that.
1:02:22And it's well-funded and they'll be around forever. Or you find that, yeah, this is actually one guy in Nebraska, the famous one guy in Nebraska. Yeah. That would be very good. But I do want to share one other technique I use to figure out how my dependencies are doing is I just use a dependency for a bit in an isolated fashion. And then I, and I will always run into some kind of issue. There's always something. And then engage with the project, see what happens. So you go to the project, say, Hey, I just put, I did it with SQLite, for example. SQLite had a new feature for database replication and it's really cool.
1:03:01I really recommend it. It was very new. And I was wondering, can I build on this? Is this solid enough? And I tried it and there was an issue and I filed an issue with the SQLite project and they fixed it in 25 minutes. And they immediately believed me that there was an issue and they were on it and it was just done. And when you do this with a dependency, you will often find out that no one responds to your worries because they're not paying attention. so actually trying something and open just and then just opening a ticket and figuring out how do how do they respond and this is incredibly telling and but again this requires work this will will require you a few hours at least per dependency uh to figure that out and then people say look i have a thousand dependencies and i won't have time to figure that out and my response would then be then you should not be shipping a thousand dependencies if you select a dependency wisely it can save you hundreds of hours of effort so maybe one or two hours spent putting the work in is worthwhile right and i think it is and there are things that i mean very like i said with cryptography i mean the the cryptographers out there they have done a better job than than you would ever do or otherwise you would be a cryptographer but it is yeah but i think it's somewhat of a lost i mean i tell people you you probably remember the npm left path uh thing which where you could left a line in the string and someone removed that or made it malicious and then the whole tower of dependencies came falling down.
1:04:37And when I tell that these days to people, I say, look, I worry about left path. Then many people go like, what is left path? Although that should be part of our engineering lore that says, look, do not go around depending on single line files that could be taken over by any random internet person someday. Are these youngsters you're talking to? Yeah, well, relatively speaking, relatively. I mean, there are many people that are like 30, and I'm starting to find people that are 30 to be a bit on the young side, but that's just me. There you go. It's all relative. It's all relative. But yeah, this is something, I mean, this is just the way modern software gets developed.
1:05:20And of course, we have to die hard to do embedded software or really the safety of life stuff. but we are in a very, we're going to, in a small minority and the minority is getting smaller. There's also new tech coming out all the time. And you've been around long enough, 30 plus years in the business to know that not all that new tech is going to be additive or useful or last. Yeah. So, so I did a talk on this and that actually for Dutch networking community, NL Nook, which is great. and the the thing i tried to come get across there was you have these these new things coming along all the time the new and shiny and there are the people that are just always attracted to the new and shiny and and they have a glorious time discovering all kinds of new things and fun and there are of course people that are still as i call it rocking it with orc uh because you get stuck with yeah this worked for me in 1997 and it will work for me in 2029 as well right um you have to strike this balance to say i i enjoy the new and shiny but not too much and you should also let go at some point of pearl or or whatever specifically pearl yeah it's high time it's high time it's high time yeah but uh and and i i talked i for myself i also sort of i'm a bit ambivalent about this because on one hand i i love the new things and and i think everyone should be so i do a lot of javascript for example uh but i've also been doing c plus plus since like 30 years and people are you could on the one hand say that c plus plus is yeah it's starting to get old and maybe you should move on to something new but there i decide no look i'm like really good with c plus plus and i'm getting the stuff done so and i know there are issues with it but hey it works for me uh but on the other hand i have no problems with doing all kinds of newfangled JavaScript things.
1:07:16And that was sort of the upshot of my talk. I said, look, enjoy the new Shiny and do study it, do look into it, but don't just shift whole frameworks every few years because they are just, yeah, the bee's knees right now. But also do not get attached to your extremely old frameworks. And I think that is actually, that's why I wrote this post and did the presentation, I think this is one of the most biggest determinants of your technical career. If you would take a look at the technical development over like 20 or 30 years, how you look at new things really determines if you're going to be in technology for the long haul and continue to be relevant and effective, or if you have to bow out and become some kind of manager because you're no longer with it.
1:08:11and and and managing that is i think a very important part of a programmer's career also yeah we're certainly in a career where you can't just learn everything you need to know at university and be done i mean if you're not interested in change and adaptation and learning new technologies this isn't really the space for you you're not you're not going to last no and but there are it turns out there are many employers that actually also do not believe in change sure uh so you you can end up in some kind of insurance place or a government tax agency and work with 25 year old technology writing cobalt on a mainframe yeah they're still rocking it over there so it's not necessarily you could come at a point in your life where you say look i don't need all these new things anymore and uh i have only so many years left in my career and i'm going to spend them in a basement uh in some kind of insurance place so it's not all bad but it is something that I think you would have to just ponder for yourself say am I going to invest also because there comes a point in your life and I think we touched on this in this presentation also where I'm just saying hey these 2 ,000 dependencies they're all bad and there is a risk that at some point in your career you go like look things used to be better and all this newfangled stuff is just wrong and when you hit that point you have to have a real good thing like, I think, well, am I leaving the industry behind?
1:09:37Am I just old now? I'm just sort of a sad person that sits there and says, well, in my time. And you should, at least when it happens, it should be a conscious decision. Where you say, okay, I love this old stuff more. And I'm not going to pretend that my old stuff is better. I'm just going to admit that I'm like really good with this old stuff. And I enjoy doing this old stuff. And I'm not going to tell you that Rust is wrong or that Swift is wrong or that whatever new thing is wrong. I'm just going to enjoy my old stuff. But it's really sad to see some people see new things happening and they just get stuck in telling you how great their old stuff is.
1:10:12So how many languages have you invested in, seriously? Yeah, so this is a terrible story. So I'm terrible at Python. So I write a lot of Python. I'm mostly for making graphs in Modplotlib and Jupyter. But my Python is really terrible. And it has been terrible for the past 20 years. because I've never written anything. I only write scripts in Python, and I've never invested in really learning that stuff. So in Python, I'm like really the copy-paste engineer. And JavaScript, I actually sort of made a weird decision. At one point, PowerDNS was like really not going well, my first company. And then I decided I wanted to invest some time in this new web stuff.
1:10:55And every serious programmer hated JavaScript with a passion. And I was like, well, that probably is the future. That was a good call. That was a good call. I'm not going to say that. I probably just got lucky at that time. So JavaScript, I'm sort of reasonably proficient in. And C++, that's a weird thing. That's my workhorse. And the thing is, I use like 20 % of C++. Because over the years, I've learned there is a good core in C++. It's like a really solid language. and it's good stuff. And then there is a lot more stuff. And I'm sort of in a sort of diet version of C++ where I'm like, look, I know the good parts now.
1:11:38I'm not leaving the good parts. I'm just staying there. So that's the other thing. And lately I've had to admit that I'm sort of skilled at HTML now, which I never saw coming. But apparently I got a little bit of that going. But that's really it. I got my start with Perl, which was also sort of, I found that people were sort of, sort of didn't take Perl seriously. And I also did not take Perl seriously. And at one point I had to do a lot of data crunching, like millions and millions of lines. And I thought these Perl weenies, I can do this better in C. And then I found out that these Perl weenies, they do know a lot of stuff about parsing files.
1:12:20And so that was a humbling experience. But these days I sort of have a trifecta going of C++ and JavaScript and mostly SQLite in the backend, because I've invested a lot of time in understanding its quirks and its stuff. And it's actually really sort of solid data handling framework. And to the point that I now sort of trust SQLite files more than text files, because it's quite easy to mess up a text file. one stray one stray carriage return and then it's done sure SQLite will not not disappoint you on that front so how do you see promising technology coming I've actually been sort of not that happy with the for example programming languages if you look at the development of programming languages you see that in the past years we've not gotten closer to the ideal programming language So there is something wrong with every programming language right now.
1:13:23So some of them, they're like really safe and some of them are really fast and some of them are really simple until you try to do complicated things. And I'm still hoping that someone someday will square that circle and have the programming language that is sort of simple and fast and safe and maybe fun. But we're not really getting there. and the other thing I of course worry about we have all these frameworks for website development and these are also not improving they're just changing and one thing that I have a series yeah one thing I would love to build if you look at any kind of software it involves interactions so you have to do things with the software and you might have to sign up for a website and then you get assets that are yours and you can configure these assets.
1:14:17And that is a whole workflow kind of system. And I would love for us to move into developing workflows instead of developing web pages. Because if you look at most interactive websites, they're still like, well, we built this page for you and that page is full of conditions. It says, well, you have to fill in this, you have to fill in that, and then you can press send and then it happens. But I would love for us to move to a higher level description where you say, look, this is a website where you can sign up for college or whatever. And this is in a domain-specific programming language where we say, look, you can only move from this state to that state once you have filled out the following things.
1:15:01And if we'd ever developed that, we would get so much more robust sign-up flows. because the system is just aware of what can happen and what cannot happen. And I would love one day for someone to develop this higher level way of interaction developing. And I'm sure that it has been done in some places. But I would love to have the kind of this as infrastructure. I would say no one has to write a transition handler anymore that decides if you filled in the right zip code or whatever. I just want a library that already knows what the zip code is as a primitive. But I don't see any of that happening.
1:15:43I see new iterations of ways of creating HTML pages. And every five years, you get a new sort of wave of these new frameworks. But I don't see any of that high-level stuff happening. I do see it happening at the super high level, almost no code. If you look at like form builders, you know, which we give to people. and they can put in their conditionals and they can say you know give me a zip code widget and put it on the screen but you know that that's like not ad hoc what's it called when it's just like specific use for that one particular bespoke yeah it does the one thing so i have this this so i also do presentations and you can hire me as a speaker and my speaker agency wrote a whole low code solution system just to do the logistics of that and it's sort of wonderful in a way because I'm really impressed that they managed to build that mostly themselves.
1:16:35But on the other hand, every time I do something with the system, it takes like three seconds to go to the next page. And I'm wondering what is it doing in the background? But it is impressive because it is built on this concept where you click this stuff together without having to be a real programmer. But the thing is, I would love to be a real programmer and have the primitives that make me do this safely and at a higher level. It's hard to resist the future. really you gotta like you've said you gotta pay attention to it but not so much that you're like engulfed by it you have to weigh it against what you know where you've been and where it may go and hopefully the wisdom you've gained gives you the ability to discern that future and how it applies to your present otherwise you get left yeah so i i had this in this talk on dealing with change, I also mentioned AI.
1:17:27And then part of the audience goes like, yeah, AI, I want to do everything with AI. And part of the audience goes like, no, it's unethical, it's bad, it's not as good, it's not reliable. And of course, it's tricky to find the middle ground in there. When you say the AI is just astoundingly good at transcribing spoken text. It is actually super human so there is this whisper system from open ai it does 99 languages and even does languages where they didn't know that it could do it and it is super human because it's better than any human being on the planning planet so so there is actually really good ai and and then of course there are things where we still have to worry what good is a model that just hallucinates stuff and and we could be yeah find that really bad but it is extremely tricky for people right now to understand that, look, we could do some wonderful things with AI.
1:18:23It really could be good, even though there are some stupendously bad things you could do with it. But I would love for AI to also come into our universe as serious programmers to say, hey, we have this tool for you. And it can recognize if a user enters something, the AI could say, well, this is probably not what they meant. Something like that, that looks over your shoulder. Or something that deals with, that you have a system that has to deal with user complaints. And you have an AI system that automatically does the triage and says, look, this user has a real problem. And this looks like only a feature request.
1:19:00I would love to have those kinds of primitives. But there are people out there that are just once so in love with AI that they're like, we're not going to need programmers anymore. and there are programmers that hate ai so much that they say we're not ever going to touch it and and adam as you said navigating that we cannot hold back the future but it's tricky to sort of navigate the stuff that is like good and that we should have more of while staying away from from the areas where we say we're going to just replace the whole thing by an ai chatbot get out of here ai chatbots yeah we've had enough of you yeah well i think that this decision tree that you've got listed, which is, you know, rudimentary.
1:19:40It's not fully featured in terms of every possible way you would decide against choosing a dependency in this post. I think that's a great example of where you'd throw AI at it or, you know, automation, let's just say like the basis of automation and AI where necessary to sort of investigate a dependency and whether or not it checks against your balance. You know, does a tech look good? I mean, maybe you could, that would be a great example of throwing ai at it like it can it can do an analysis of a code base and say well what does good look like and then the the model obviously depicts some of that stuff who writes it that's a pretty easy one to just automate but that through databasing but you know there's a lot of ai you can do here with this decision tree to to make choosing dependencies slightly faster and easier yeah i have to admit i i had not thought of that yet but um people did mention it they say look, I have 1600 dependencies and I'm going to make the AI look at them.
1:20:39Yeah. It's not necessarily the worst thing to do because the AI might, in fact, very well just figure out that, look, this is just wrong or this code is already in North Korea. So, yeah, no, it's actually, as a layer, it is probably an interesting thing to do. Well, I mean, just if you've got that many dependencies, is it's literally impossible to investigate your own dependency tree. It's better than nothing. It's one of those like, what do you got to lose besides maybe a few cents? Well, Bert, what else? Anything else you have on your mind that we haven't talked about yet? No, no, I think we touched on the, yeah, we spoke a lot about dependencies, of course.
1:21:19Of course. And yeah, the thing I keep harping on about, about how do you select the new things that you like, that you don't like? That's something that I would really recommend people to spend a bit more time on because it's really strange. If you look at how we choose what technology to use, it's actually quite often a sort of random thing. It's like, hey, I found this stuff on Hacker News and I liked it and I tried it for a bit. I liked it as well. And then accidentally we pick the future direction of our career or of our software project. So I would really recommend people sort of making more of a conscious decision on these kinds of things.
1:22:00But we covered that already. Yeah, I think that's a really good point. I think we do tend to overvalue social proof, you know, what everybody else is doing, and undervalue our own ability to dig in and make a decision based on what we find. Yeah, one of the things I stressed in my talk also on selecting these technologies, You can use your coworkers or the people that work with you on the project and just tell people, look, I'm going to present on this new thing. I found this new magical database and it does 300 million lookups per second. And these are my experiences. These are the graphs that I made.
1:22:38And then your own coworkers could be the filter because they might not be impressed by your story, for example. or you might find because you actually have to get down with it to actually make the graphs to present to your co-workers that you find that the graphs are actually not that great and so in this way you can both entertain the people around you with with your findings and also use them as a a way to keep yourself honest and if you don't have any co-workers around you can write a blog or a vlog or go in a podcast or whatever but whatever forces you to take a serious look at something and work on how, how would I present that to someone else?
1:23:18I think if you do any of those things, I would recommend going to changelog.com slash community. That's a good place to hang. You know, share your findings there, share your thoughts there. There's a thriving Zulip there waiting for you to thread and be a part of, and it's free too. Bert, what's the best place for folks to connect with you on the internet? Yeah. So that's an interesting one. So I've been pretty enjoying my time on Mastodon. I think it's really a geek heaven. So that works really well. But interestingly, people ask me that question quite a lot. And I found, and I think it's very weird or strange sort of regression, many people are now running mailing lists again and sending out updates and newsletters and stuff.
1:24:05And I found it to be a bit pompous to write updates on what Bert has been doing. But apparently there is demand for that. So I'm probably going to restart a small newsletter, that kind of stuff. I also used to tell people that just follow me on LinkedIn. But the problem there is that on the one hand, I'm sort of active in Dutch politics things, which are entirely not interesting to the rest of the world. To a lot of people, sure. Yeah. And the other way around, and there are many people there that follow me for the Dutch politics stuff. And they're sort of confused. They are massively confused to suddenly see C++ and then they go like, what is this?
1:24:49So actually for the newsletter thing I'm building now, it's actually going to have like four newsletters. It's going to be one for people that like Dutch news, that like news in English, that like stuff about politics and that like stuff about technology. And you can pick your own quadrant in that to see which updates you want to receive. But for now, if you go to my website, and I'm sure it will be in show notes or whatever, or you can just Google or search Bert Hubert, and then I have a website which is RSS feeds and stuff. Nice. Gotta love RSS. Here's a name you can name your new newsletter.
1:25:30You can call it Bert's Pompous Update. Just go right in there. Bert's self-important things that you actually all asked for. Yeah, just tongue in cheek that sucker. You know, it always works. Cool. Love it. Appreciate you coming on the show. Nice meeting you. Adam, anything else you wanted to talk about or ask? Keep that software simple. Yeah, that's the goal. That's the goal.
1:25:54Keep that software simple. It sounds so easy. So achievable. Well, it ain't easy. That's for sure. But it is achievable. If you follow some of the advice that Barrett shared with us on this episode, We'd also love to know your thoughts on building software that lasts. Let us have them in the discussion thread, which you'll find in our Zulip community, which you can join for$0 at changelog.com slash community. Let's give one more shout out to our sponsors of this awesome conversation. Thank you to Fly at fly.io, to Retool, retool.com slash changelog, to Temporal. Find them at temporal.io. And of course, to DeleteMe, head to joindeleteme.com to get started.
1:26:37And thanks to our Beatmaster in residence. It's the Beatfreak, Breakmaster Cylinder. We love you, BMC. Oh, and we have a little bonus for our favorite kind of listener. That's a Changelog++ listener, of course. Stay tuned, friends. We explore what a dependency-focused AI might look like for 10 more minutes or so. All right, that's it. This one's done. But we'll talk to you again on Changelogin' Friends on Friday.
1:27:10Thank you.
1:27:47Game on.
From the publisher
After 30+ years in the software industry, Bert Hubert has experienced a lot. He founded PowerDNS, published articles for places like IETF / IEEE, and built his own parliament monitoring system. That just scratches the surface.
Recently, Bert wrote about what it takes to build software for the long term. Let's dig in.
