The best, worst codebase (Interview)

30:30Like it was absolutely insane. There were eight local police and two secret service agents and they don't take the backpack full of hard drives that was sitting right next to the computer, which is just like top notch police work. Yeah. And then the second was we had gotten an iMac at that point. And I watched as two local police looked around and they're like, where's the rest of it? And one had to, another guy to be like, I think this is the whole thing. That's hilarious. It's just all in one. It was great. So yeah, I didn't do any real hacking. I just found something unauthenticated. It was not complicated.

31:08I was not some massive hacker. just uh i ended up getting interrogated by the company for like eight hours they just dismissed the charges or how did it yeah no charges were ever filed it was just a search warrant you know pending charges based on what they found which of course there was nothing i didn't do anything i just found a security vulnerability but my mom made me tell the company because she was worried i wouldn't have a job or whatever and i knew they didn't know i worked for them which was true they were a very disorganized company but i ended up getting interrogated by some lady from corporate who when I asked for a lawyer to sign documents, she stood up and screamed that I threatened her and I got fired, not for anything I did there, but for threatening her.

31:50Wow. Yeah. So that story plus Miss Johns together got you this job. Because they're like, well, you must be good at what he does if the Secret Service is after him. Yeah, yeah. I don't think she even knew the, you know, like technical details or whatever, but it was enough they were intrigued that they let me get an interview and luckily uh yeah i got the interview i think six weeks into my internship i got hired on full time and yeah were you actually good i mean you know i could fake modesty but yeah i was good i mean i wasn't good compared to now right like i was awful like if you look back at all of that stuff i was terrible but you know for where I was at as a junior developer, right?

32:36Like as a, like basically like a recent grad, I knew my stuff, right? I was able to, like when I joined, uh, we took the backlog from like 60 items in the queue that had been there. And I, I was able to get like 40 of them solved pretty quickly. I mean, that's why I got hired in six weeks from intern to junior developer. It turns out like all the stuff I had been doing in my spare time was actually real software. I just didn't realize it. Huh. That's cool. Yeah. He's the kind of kid who has got a backpack full of hard drives. Of course he was good at what he did. You don't just accidentally end up with a backpack full of hard drives.

33:14I really didn't. And I think I'm sure like part of the reason I want to share this is not like to brag about myself. It's like I, because I grew up, you know, like none of my family had gone to college. It was a very, you know, working class family. Like I had no concept. that this was something I, you know, I read a bunch of tech articles, I'd seen all of this stuff. And I knew some people out in California did this. But as a kid growing up in a small town in Indiana, I just thought what I was doing couldn't possibly be the real thing. And so like, I was two years out of school, I'd never once considered like programming as a job, because like, I just didn't think I was good enough to do it.

33:52Right. Then you find this code base, and you realize how many hundreds of thousands and millions of dollars in labor have gone into this monstrosity. Exactly. And I've realized my worst code is, I write bad code. I wrote tons of bad code at that company. Zero question. I mean, they had to scrap a whole project that I did. I made all sorts of mistakes. But you can not be perfect and yet contribute quite a bit. And these people, yeah, they created value for the company despite this code being awful.

34:36Okay, friends, here are the top 10 launches from Superbase's launch week number 12. Read all the details about this launch at superbase.com slash launch week. Okay, here we go. Number 10, Snaplet is now open source. The company Snaplet is shutting down, but their source code is open. They're releasing three tools under the MIT license for copying data, seeding databases, and taking database snapshots. Number nine, you can use PG Replicate to copy data, full table copies, and CDC from Postgres to any other data system. Today, it supports BigQuery, DuckDB, and MotherDuck with more syncs to be added in the future.

35:18Number eight, Vect2PG, a new CLI utility for migrating data for vector databases to Superbase or any Postgres instance with PG Vector. You could use it today with Pinecone and Qdrant. More will be added in the future. Number seven, the official Superbase extension for VS Code and GitHub Copilot is here. And it's here to make your development with Superbase and VS Code even more delightful. Number six, official Python support is here. As Superbase has grown, the AI and ML community have just blown up Superbase, and many of these folks are Pythonistas. So Python support expands. Number five, they released log drains so you can export logs generated by your Superbase products to external destinations like Datadog or custom endpoints.

36:06Number four, authorization for real-time broadcast and presence is now public beta. You can now convert a real-time channel into an authorized channel using RLS policies in two steps. Number three, bring your own Auth0, Cognito, or Firebase. This is actually a few different announcements, support for third-party auth providers, phone-based multi-factor authentication, that's SMS and WhatsApp, and new auth hooks for SMS and email. Number two, build Postgres wrappers with Wasm. They released support for WASM WebAssembly foreign data wrapper. With this feature, anyone can create an FDW and share it with the Superbase community.

36:50You can build Postgres interfaces to anything on the internet. And number one, Postgres.new. Yes, Postgres.new is an in-browser Postgres with an AI interface. with postgres.new you can instantly spin up an unlimited number of postgres databases that run directly in your browser and soon deploy them to s3 okay one more thing there is now an entire book written about superbase david lorenz spent a year working on this book and it's awesome level up your superbase skills and support david and purchase the book links are in the show notes that's it super base launch week number 12 was massive so much to cover i hope you enjoyed it go to superbase.com slash launch week to get all the details on this launch or go to superbase.com slash changelogpod for one month of superbase pro for free that's s-u-p-a-b-a-s-e.com slash changelogpod

38:05you didn't introduce the calendar did you no yeah so this is uh there was a calendar table which was literally a hand filled in calendar that my understanding at best was that when contractors logged in versus when employees logged in there were certain days that employees were allowed to log in but contractors like on weekends and holidays could not log in. It was supposed to be completely forbidden. And this was the system that was doing it. And so one time, they filled it in for a few years and it actually ran out. And they had to scramble to figure out how to log in and then just had an intern fill in another five years.

38:47Had no idea which. There were so many services, so many programs running in the background, so many cron jobs. They had no idea which one had been locking people out. I love it. So just filled it in more. Just one row per day and you just got to fill it out. And it's going to check against that row for the day. And if there's no row, then they can't log in. Exactly. I also had a task that I don't mention in the article where there was this like 5 ,000 line Pascal program that had been apparently like very mission critical for the last eight years. And all of a sudden had started failing. And they had no idea why.

39:23and I was told to rewrite it in C Sharp because they didn't want to support Pascal anymore. I looked at this thing. It was just, it was only 5 ,000 lines because it was copied and pasted. Like they unrolled a loop by just like copying and pasting code over and over again. So my end code was like 200 lines and I get it. I make sure that like everything I'm seeing is like identical. I was like really thorough to make sure I got the program right because apparently it was really critical. I was given a week deadline. Okay, I get it. We put it in service. And what it was doing was sending a bunch of emails about some process or other.

40:03I didn't know the details, but reading from a table, sending emails based on reading from the table. Not complicated. And I immediately, the day I turn it on, my manager comes over to me and says, please turn that program off. We are getting constant emails from people. Apparently, this program hadn't run in eight years. It was about something that was eight years old, and it was spamming people's emails every half hour because it couldn't find the data it was expecting. Wow. So it was defunct, basically. It had been not running. It had been running incorrectly for eight years. They just finally turned on alerting, and that's why they started noticing that it was failing.

40:44Oh, okay. Yeah. So you did implement the correct behavior. I've implemented it exactly right. It's just that, yeah, nobody knew that it was failing because that program, that part of the business had been gone for eight years, but those people were still there. Who knows, man? That's wild. I think the other one that I loved was there were whole programs in source control that were just decompiled sources. so they were c sharp but they had lost the the source code to them so they just took the binary ran them through a decompiler and checked them into source control okay and you had to make changes to this application and i don't know if you've ever worked with like c sharp decompilation but it is imagine it's unreadable right it is completely completely unreadable it is compiler output.

41:44It's like if you had to work on JavaScript minified code and I had a person, one of these was our time tracking system. The way we tracked all employees was a custom built application, but we had lost the source control, lost the source code. So it was decompiled. And there was some list that was wrong according to a business person. So I go in there and I'm looking at this like Boolean logic that's been like optimized by a compiler. And I finally like get it i write down like the logical formula and i plug it into wolfram alpha and it it tells me like the simplified form oh cool it was great i was so proud of myself for thinking of doing that because it was really complicated it turns out to be like this and that or this and that or this and that that was like the whole entire thing and i figured out what each of these variables came up with i was so proud of myself i go to the business person i'm like all right here's all the variables here's the Boolean logic, you know, what does it need to be instead?

42:41And she's like, oh, uh, I don't know. I don't know what any of those variables mean. I don't know what any of these terms mean. Instead, I had to, I had to go change a variable, print out a new list on a piece of paper, bring it to her desk and she would mark which ones were right or wrong. Oh, wow. That's in check. And then I would try to figure out based on her marks, what the bullet, and we, we did it it took like you know 10 rounds of printing out pieces of paper and letting her mark on them but uh yeah and then did you decompile that and throw it in source control or like how how did you how did you recover from the circumstance or did you just perpetuate it yeah i mean there was no you just now edited that sort that was the source code right that's the only thing we have is this crazy decompile thing i cleaned up that little bit there made it a little cleaner and added some human readable terms to it but there's no way you can fix it was probably a 30-40 ,000 line application no way you're going to rewrite all of that in the time given why did you stay here?

43:46why did you keep doing this job? was it just like a weird conundrum slash challenge? like this cannot be real and I must stay to see what happens I mean you gotta realize I didn't stay there super long but that was, I probably would have, but I met my now wife and moved. But like, I mean, there were tons of people who'd been there five, six, eight years. This was one, a small town, you know, with a big city near it, but like there's not a ton of programming jobs and all of them are kind of equally crazy. Like I knew people who had worked at other companies now, you know, and come here. And I think this kind of stuff exists a lot more.

44:27I mean, you know, there's tons of comments, right? Being like, I thought, just like we had, you know, here, it's like, I thought I worked on this code base, right? Like, this sounds very familiar. But also, like, I didn't know any better. I just assumed, like, this is what code looks like, actually, in the real world. You know, I'd seen some open source stuff, but never really dived into it. But I'm like, oh, maybe open source is better. But a company, this is just what you have to deal with. And while it's definitely the most story-worthy code base I've worked in, all of the things that were bad were just so obviously bad, it was not a bad place to work.

45:04I would actually rank it on one of the better places I've worked. Not the best, but one of the better places I've worked. Part of that was definitely my position in there. I had a great manager. He was just so supportive, so nice, always made sure that I got the growth opportunities that I needed to become a better developer. He saw the potential that I could do and made sure to help me and get more senior developers to help me learn stuff and challenge me. That was really good. But also, this will show my very strong bias here that I know you all might not necessarily agree with, but there was no agile process or none of that stuff, which I have found to be like the main factor in job dissatisfaction for myself.

45:54So like the lack of process was so freeing and so nice. Yeah. Well, you're not going to get a disagreement from me on that one, but maybe. We don't do agile around here. We do whatever we want, basically. We do code stuff. Yeah, I wasn't sure. You know, I don't know your exact thing, but like with the Kaizen and all of that, you know, obviously. Well, the Kaizen just means continuous improvement. You know, that's something that I'm sure you're into, right? Like let's make things better all the time. Yeah, I do like, I mean, your Kaizen episodes are great. You know, just wasn't sure. I know it's always contentious when I say like, not a fan of Agile, even with a lowercase a, just not a fan.

46:30So it was a good company to work for. You know, the biggest problem really, why I wouldn't have stayed there longer is like being underpaid. You're in a small town. There's limited talent. There's limited places that you can go though. So it can pay you way less. It turns out like I was making the same Famous people who had been there like eight years, which is just, and who had job titles way over me, which was just wild. Wow. But this is programming in the Midwest, you know? I know tons of people who are at companies like this with equally crazy code. I mean, even like Salesforce has a branch here in Indianapolis and all the code I hear out of Salesforce sounds similarly wild to this.

47:10Mm-hmm. Well, when you have so many people and so much, I guess, momentum, you have to make progress. and sometimes progress is like just duct tape that part you know and that's kind of what like the employees table this seems like duct tape like why in the world do you have to drop this table at 7 15 every morning and then repopulate it with a new injection and then people can't log in like why is that the way or the same thing with the sales numbers like why do they have to like claim these wins and then put it on this board and they were able to subject these interns to doing all this minion work basically to get their seemingly their numbers projected properly i don't know it's hard to decipher exactly what's happening there but yeah yeah it's a little complicated but yeah i mean it is the basic idea for that one just to be clear is like sales people you know obviously there's like the real accounting i know there were some comments on hacker news of like that just sounds like fraud there's like the real accounting system and then there was the rewards system where they get bonuses and stuff this was the reward system and you know yeah they were trying to get bonuses every month and if they made a big sale at the end of the month and they had already gotten their max bonus they would just move that to the next month so they could get their bonus for next month right so they had a cap on commissions and things like that and this was moving it around that's funny because i was encouraged when i was in sales back in the day like hey let's just move that sale to next month because you've already reached your quota this month like good good for you yeah and it got encouraged here but the way that it was implemented was interns manually writing SQL statements.

48:46Take that hourly wage out of your bonus. It costs a little extra. There's some intangible cost there on that bonus system. There were people who the whole internship they were there, they never got to write any code other than these SQL updates. And I think that's a shame. That's not how you should treat your interns. But that was one of the things. I just refused to do it. I just never wrote one. Every time it would be assigned to me, I would just go find something else to work on and do that instead. I wasn't a great employee also, to be clear. I was 20. I was a little bit more prideful than I should have been, a little bit more arrogant than I should have been for sure.

49:31But yeah. What about these hard drives and guilfoyles? Is this a Silicon Valley nod? or is this a real Guilfoyle? It can't be really a Guilfoyle, right? No, no. It's definitely a Silicon Valley reference. Okay, good. Yes, yes. I did not do it as bait for this show, but I guess in hindsight, I should have thought of that as a benefit too. Yeah, it definitely was like, when I saw that, I thought, Adam's going to love this. There's someone named Guilfoyle? Well, you say, let's call him Guilfoyle. So I figured, and then I immediately command F'd and typed in S-I-L and found nothing for Silicon Valley.

50:04I thought maybe at least referenced it. Like, let's, you know. No, I figured it for anyone who knows, you know, is a good reference. And anyone who doesn't, it's a good enough name, right? It's kind of a fitting name, I feel like. Sure. So, yeah, no, his actual name, I never, I wouldn't, I felt a little awkward, like, using his actual name. Because I never met him. I have no idea who this guy was other than through his code. And so, yeah, he was, he now, I don't think, I looked him up. I don't think he's a programmer anymore. he sells exotic pets. So yeah, I thought Guilfoyle just seemed like a fitting name, especially if you know the reference, right?

50:44It was, that was always the sense that I got of this guy. And yeah, he was, I mean, the most prolific programmer I have ever seen. The amount, the sheer amount of code in that code base, that was his. And the sheer amount of like applications that random customers or customer support people would have that were just from scratch Windows applications that he wrote with complicated logic. Any time, because he refused to use source control, which was why we had his hard drives raided on Munch's desk, he refused to use source control. If a person asked for a code change, a feature change, he would just rewrite the application from scratch.

51:26And he was apparently, from everything I heard, that fast that pumping out a brand new like 10 ,000 line application in a day was like not unheard of for him. But the code. This is like a legend. This is like a legend. Yeah, total legend. Yeah, absolutely. And like, yeah, I don't, because he didn't use source control, I have no idea how fast he actually wrote this code, right? Like, so you don't get any history on it. But like. That's how the legend continues. You know, he can't have a trail. No paper trails. Yeah, I wrote this in a day. Maybe he just anticipates needs way in advance, right? or puts his code through some obfuscation every single time.

52:04So it looks like it's slightly different, right? I don't know what it was. But yeah, his code was, it was a trip to try to understand though. There was just every time, there was never a consistent pattern to the craziness. It was like he just woke up every day and thought, what new weird programming pattern can I abuse here to write this application? services that were pure functions that like i literally do not understand why they existed clients that were just like these these super thick clients that like i mentioned the article completely empty classes classes i i did not exaggerate in the article they were empty classes they had a class definition method definitions but there was no code in any of the methods and it was it was all to like build up a structure and they're like you know hierarchy would go like 10 layers deep of inheritance.

52:59And it was all to build up the structure that then would become a pipe delimited string sent over a socket. What? That was all driven by the database. And so like, when you looked at it, you were like, okay, what is even happening? Like once you even figured out like, oh, this is a data structure, not a class hierarchy. It was like, wait, what do they turn into? Oh, well, it's like dynamically choosing which column of which table to go grab this field from. And now it's custom there. And then the table would encode like, How do you encode the value? So you could infinitely configure how the delimited string would be created.

53:34But there was no reason to do that because it's an old application that hasn't changed in 15 years. And the same message was written every time. It was wild stuff. But I actually debugged that bug for a year. It manifested itself as like 15 different cases in the system where things would just go wrong. And I thought it was like a memory leak for the longest time. I thought it was all these things. And like finding out that it was just some legacy third-party application reused unique IDs every month was like the most exciting and most letdown bug find I've ever seen in my life. Yeah, not exotic.

54:18I thought for sure it had to be Guilfoyle's fault, right? His code was too clever. I really wanted to blame him on his cleverness. I was going to say, it's convenient sometimes to have a Guilfoyle. It's like a patsy. When something's wrong, you've got someone to place the blame on because he's been prolific and he's done all these things and he's not around, so surely. Yeah. Gosh, Guilfoyle, what's wrong with you? But no, this was a third-party system. What about ops in this case? You're talking about the code base, but somebody's got to keep that database up and it seems like he's getting hit in wild ways.

54:52Like this chain function, for example, it's probably got the database spinning. The disks are spinning. And I'm assuming that's the day of hard drives. Yeah, so the database was definitely, we had quite a few DBAs. The DBA to programmer ratio was pretty high. And we had some very beefy machines running this SQL server setup. On-prem? On-prem. Obviously. So yeah, we had everything. We had a, like actually at the office I was, there was like a data center area as well as like a shipping area. So it was on-prem, it was local. There was some talk about like the company building a private cloud system and things like that.

55:34But, you know, it's a little early for them to actually do that. Nothing ever came of it. Yeah, there was definitely some beefy things. Most of Ops, though, really ran through one guy who was really good at his job. He was the Ops guy. I mean, there were technically other ones, but everything I ever dealt with, it was him doing it and maybe delegating some tasks occasionally to other people. But it was a pretty bespoke kind of setup. Deployments were all done by hand by him late at night, so that way it wouldn't affect anybody. you know it was it was that kind of place that kind of setup where you know servers had pet names and and all of that right it was well before there was some early maybe looking at puppet maybe doing some of that but nothing really came of it so yeah it was a it was a pretty this is this is honestly one of the things that like was so strange is it's the scale of the actual code base, the scale of how many people are using this is small, but not completely trivial.

56:44And yet, this legacy code base, especially, the other applications that were the big rewrites had not seen production use. And we were able to, with me, an analyst, a QA, one senior developer, and four interns, were able to out-compete all of this rewrite where we were adding new features, fixing old bugs, and doing all of that in this system while they were off doing their big agile processes and doing all their story pointing and all of that and never getting anything done. So it was fun. When you actually think about it, if you're not one of these big web-scale companies, servers are simple.

57:31Code's not that complicated. even if the code's complicated. You can make changes if you just don't get in your own way. So yeah. It sounds fun. I mean, I would like to work on this code base for maybe like a month and then move on. But I'd like to visit. As a game, maybe. Yeah, it is a game. And I mean, it's got to feel like that to a certain degree. It did. It felt very much like a game. But I think that's how I approach most of this work. Like I said, I think the job I'm at now, we've got a big, massive old code base. I now work on a fork of RhinoJS. Okay. I don't know if you all remember Rhino.

58:08I do, but I can't remember what it does, but I remember RhinoJS. It was like involved in my cappuccino days. I think they were using it back with Objective-J. Yep, yep, that sounds about right. It is a JavaScript implementation written in Java. So it is a compile to JVM bytecode and an interpreter all written in Java. It was abandoned years and years ago, but I now work at a company that has a long-term fork of it that runs millions and millions of lines of customer JavaScript. Really? And it's got some custom features. Say more? Sounds good. Yeah, so we're now trying to refactor away from it, But like, for example, in our original version of JavaScript that people still use to this day, if you did dots, it was like doing question mark dot.

59:08So if it was undefined, it wouldn't throw an error. Okay. Just return undefined. This was a choice by the founder of the company. Like, I guess I just have a knack for finding these code bases. I don't know what it is. Where things are just crazy. and I think a lot of developers work in these kinds of things, right? They just don't talk about them publicly. Totally, totally. I've had similar experiences, all I think at smaller scales, both in terms of company size and code base. My craziest one was I inherited, I did a rescue project for a boat shop somewhere in Georgia where they just needed, like basically it's a Ruby on Rails application that ran back office for a boat shop.

59:52A lot of merchants, a lot of sales, a lot of this kind of stuff. So similar tables and stuff, which is why it's probably resonating with me. And they had lost the original dev team. It was like a contract team came in, built this system and left. And then the IT guys who are also third party kind of took the system over because they were just nice guys who are helping out the company. That was a successful boat retail shop and relied upon this application to run their business now. And this was like really early Ruby on Rails day. I think it was like version 1.2 or something. And so there's a lot missing.

1:00:24And this team came in and wrote a lot of very clever code, basically implemented a meta framework on top of it. And it was insanity. It took me a very long time to unravel. And then they left, you know, and these people were left kind of high and dry. And so I was happy to help out. And I had the challenge, the game, like all these things. There was no development system. It was like production. And I copied the code down and tried to get it running on my machine, you know. So you're very much like doing crazy stuff, very small increments at a time, trying not to break things. And so that experience just resonates with everything you're saying, like the meta programming, specifically when you're talking about the thing that generates data structures from classes and the database kind of is the programming system as well, if you want it to be, but no one's using it.

1:01:14Like all that stuff was there. And it took me a very long time to be able to unravel it and understand it to the point where I was like, oh it's kind of clever once you know how it works but that's terrible thing and so i think and i didn't you know i didn't write up about that at all i think there's tons of code bases like this out there yeah i wish i had finished this write up i i haven't actually done much ruby although i did work at shopify on yjit the the ruby jit compiler okay a little bit before getting laid off sadly uh you know it happens but the one ruby code base i worked in there was just this little bit of like super clever metaprogramming which of course ruby loves you know lots of people love their metaprogramming and i i could never find the reason i never wrote it up is i could never find a non-circular starting point every time i would want to explain something i would have to like it would because the code like meta looped on itself every single time there you go see that's the problem yep it's like a time travel movie which was we talked about pre-show so not a Good callback, but yeah, yeah, yeah.

1:02:19That's one of the problems with metaprogramming is just too meta sometimes and you can't unravel it. Yeah, I mean, I think this is, you know, not to self-promote or whatever, but this is one of the reasons that I really enjoy the Future of Coding podcast, because I think there's a lot of code out there that people aren't happy with, and I don't think we talk about it. In some ways, I think looking at this legacy code base is the same thing for me as looking at the shiny new stuff. I think oftentimes we don't appreciate code for what it is. We don't look at what's come before. We kind of look at legacy systems like they're all bad and there's nothing good to gain from them.

1:03:01I've seen a bunch of blog posts talking about crazy code, but one of the things I wanted to do in this article was talk about crazy code, but in an endearing way. I loved this crazy code. I just love code. I think no matter how bad it is, it's one of those things that code is a medium to put information down that is unlike writing. You can get a sense from this application, like from my blog post, what writing code in this code base was like. But like you said, go and do it for a month. It's fun. It's interesting. And I think there's so much more code out there and so many different ways of writing code that we just haven't really unlocked yet that I want to see us do more of.

1:03:44Yeah.

1:03:50Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing, docs, and more. and they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with Pydantic models for requests and response objects and HTTPX client for async and synchronous method calls and support for server-sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com slash Python.

1:04:35Again, speakeasy.com slash Python. And I'm also here with Todd Kaufman, CEO of Test Double. TestDouble.com. You may know Test Double from our good friend, Justin Serles. So Todd, on your homepage, I see an awesome quote from Eileen Yucatel. She says, quote, hot take. Just have Test Double build all your stuff, end quote. We did not pay Eileen for that quote, to be clear, but we do very much appreciate her sharing it. Yeah, we had the great fortune to work with Eileen and Aaron Patterson on the upgrade of GitHub's Ruby Rails framework. And that's a relatively complex problem. It's a very large system.

1:05:15There's a lot of engineers actively working on it at the same time that we were performing that upgrade. So being able to collaborate with them, achieve the outcome of getting them upgraded to the latest and greatest Ruby on Rails that has all of the security patches and everything that you would expect of the more modern versions of the framework, while still not holding their business back from delivering features, we felt was a pretty significant accomplishment. And it's great to, you know, work with someone like Eileen and Aaron because we obviously learned a lot. We were able to collaborate effectively with them.

1:05:48But to hear that they were delighted by the outcome as well is very humbling for sure. Take me one layer deeper on this engagement. How many folks did you apply to this engagement? What was the objective? What did you do, et cetera? Yeah, I think we had between two and four people at any phase of the engagement. So we tend to run with relatively small teams. We do believe smaller teams tend to be more efficient and more productive. So wherever possible, we try to get by with as few people as we can. With this project, we were working directly with members from GitHub as well. So there were full-time staff on GitHub who were collaborating with us day in, day out on the project.

1:06:28This was a fairly clear set of expectations. We wanted to get to Rails, I believe 5.2 at the time, and Ruby like 2.5. Don't hold me to those numbers, but we had clear expectations at the outset. So from there, it was just a matter of figuring out the process that we were going to pursue to get these upgrades done without having a sizable impact on their team. A lot of the consultants on the project had some experience doing Rails upgrades, maybe not at that scale at that point. But it was really exciting because we were able to kind of develop a process that we think is very consistent in allowing Rails upgrades to be done without like providing a lot of risk to the client.

1:07:10So there's not a fear that, hey, we've missed something or, you know, this thing's going to fall over under scale. We do it very incrementally so that the team can, like I said, keep working on feature delivery without being impacted, but also so that we are very certain that we've covered all the bases and really got the system to a state where it's functionally equivalent to the last version, just on a newer version of Rails and Ruby. Very cool, Todd. I love it. Find out more about TestDouble's software investment problem solvers at testdouble.com. That's testdouble.com, T-E-S-T-D-O-U-B-L-E.com.

1:08:01It's very easy, especially when you come into a code base, to, you know, Guilfoyle the thing, in the sense of how I was talking about Guilfoyle earlier, where it's like some other who was dumb or incompetent or malicious did this. But as you actually start to like work with it and talk to people and learn about it, okay, there are politics and there are power struggles and stuff. There are things that happen because we're humans. But a lot of it is like they made the best decision they had at the time with the information that they had. And I'm staring at it with completely different perspective years later.

1:08:36And like that stuff is fun to learn. and you actually realize like, yeah, that duct tape was really reasonable considering all the things they considered. I just don't know those things. And so I really do appreciate code from that perspective, especially legacy systems that are still powering businesses and bringing value to people is that we want to be smarter than everybody else, but those people just had different contexts lots of times that we just don't have. And you start to learn those contexts and it gives you a new appreciation for the code that you're looking at. and that's cool. Yeah, Peter Nauer of Bacchus Nauer form, so BNF.

1:09:14Peter Nauer actually has this great paper called Programming as Theory Building and one of the things in there that I think is really interesting is he talks about code bases dying and by that he doesn't mean that the code isn't running anymore or no one makes changes on it. He means that the people who knew the original context of the code base are no longer there. They're no longer working on it. And so everyone else that is having to make changes to this code base is working on a dead code base that's no longer alive because they've completely lost the theory of the code base. Why is this here?

1:09:51Why put these lines in the way that they are? How do I make these kinds of changes? And he argues in there that you can't revitalize a dead code base. Once a code base dies, you can create a new theory about the code base, but it's impossible to ever recover the original one. And I feel like I've worked a lot on dead code bases where the people who, yeah, the people who knew that context once are long gone, don't work at the company anymore. And in some ways, like this code base wasn't dead because of Munch. Because Munch could always provide that like little bit of context, right? Breathe some life back into it.

1:10:33And had Munch not been there, the code base would have been completely dead. No one would have had any idea what was going on. Is Munch still there? I haven't checked. I don't know if he even has a LinkedIn or anything. Maybe he does, but probably hasn't updated it, even if he did. So I'm not sure. I know the company now doesn't have that name anymore. I don't know if that, you know, I don't know if that code base is still running or not. You know, it might be, but it also might, because it was customer support and sales, that when you know you get a big merger like that that's one of the things that often gets like they just choose one right credit card processing code i'm sure is still running but customer support might not be code that code base might literally be dead now merchants might be gone merchants three if they ever got to that point right might not have any more data and i don't know yeah i do like the way that you compliment it though you talk about the the beautiful mess that section there where you talk about which is kind of what you're talking about here where you there were no overarching design systems to work in there was a lot of freedom there was no documented design system you mentioned that there was any concerns of co-duplication were out the window you can sort of carve out your own little section because trying to fix the big mess was impossible so you just gave up and just worked in your own little world of insanity, as you say here.

1:12:02I think that's kind of cool that somehow, some way in this mess, you found beauty to enjoy. I think it's something that we should do more of. I've worked in systems since that are a mess, but everyone's always trying to fix it. I get that impulse because I hate the mess too, but sometimes it's just beyond fixing. This is why people give the advice of don't do the big rewrite because it's much bigger than you think it's going to be and it's probably going to fail. And I think the same could be said of trying to make sense of a 10-year-old, hundreds of thousands of lines code base. There's just no way you're going to be able to hold it in your head.

1:12:49And when you try to come up with some overarching scheme of what the code base is really doing, and now I can come up with the perfect abstraction and it will do all of this. I just think it's a losing prospect, right? It's just not going to work. And so, yeah, I think we should embrace more, even in good code bases. I think we often want uniformity. We want everyone to have the same coding style. We want everyone to follow the same coding rules. And I've found that often that, to me, that ends up causing more problems in the long run. because once you have to have everything consistent, as soon as there's a big change that needs to be done, it actually becomes harder because you can't do the big change all at once.

1:13:33And if you require consistency, it's harder to do those small changes each time. Even the connection to the users is interesting where you put in the after section where you describe it as a ragtag of juniors, essentially. All the serious senior folks have gone away. even like we mentioned this as a game it's kind of interesting to think about this like as a thought experiment of this ragtag of juniors just like figuring it out talking directly to the users talking directly to the support rep who's got the problem how can i make your life better to me that's like in the grand scheme of things it seems kind of interesting because i said hey why'd you stay there and you're like well because and i think i can kind of see that light now in the after section.

1:14:17Yeah. And the next job I took actually had the exact opposite thing where we were actually completely forbidden from ever talking to our users, not just as developers, as a company. I worked at a big, a company that was a, it's a big private company. They buy a bunch of companies and we were building an application for a sister company in this big group. And they hired a contractor to be the go-between. And they were scared of us ever talking to end users because they were worried those end users would think that their job was in jeopardy if we were rewriting this application. So we were never allowed to talk to them.

1:14:58And I worked at that company for a little while. And then I left and then went to a startup that didn't do very well. And I came back as a contractor at the original company. And at that point, they had changed this rule where they now were allowed to talk to their users indirectly, not directly. How so? Well, okay. So there are actually now users who are in the exact same building, two doors down from where the developers were, but they couldn't go talk to them. What they could do was those people would write three by five cards of feedback and they would post them on the wall. And I came back as a contractor and I was so interested to see what is the feedback on the system that I knew was not a great system.

1:15:44It was a completely greenfield, brand new system. And because of organizational issues, as you can imagine, it did not meet user needs. And I looked at a few of them. They were all like, please fix this UI element. Please do these things. But one of them will always stand out to me. And it was very simply put, it was, remember, you're supposed to make our lives better, not worse. Wow. That one hits hard, doesn't it? Yeah. Yeah. And that's what I had done. The whole application I had been building with this team of 30 people was making these users' lives worse. That sucks. I mean, that's... that's not how it's supposed to work.

1:16:34Yeah. Yeah. And so like, that's what I, you know, looked at like the contrast between this first job where, yeah, things were an absolute mess. Whereas there, I mean, there was no tests, right? Like they just literally didn't exist of any sort, not unit tests, not integration tests, not anything. We had manual QA that sometimes did some things. I had one really good QA for a while. She was fantastic, but like, you know, by all standards, it was wrong. Whereas the next job, by all standards, it was supposed to be good. It was right. You know, it was, you know, yes, everyone has different opinions, but it was spring and it was angular and it, we had a hundred percent test coverage.

1:17:16That was the rule. Every line, every, if everything had perfect test coverage. We had a end to end test coverage suite. We had dedicated technical QA that did all of this. And yet it was a way worse code base. It was plagued with constant problems. It didn't meet the company needs. It didn't meet the end user needs. It couldn't scale. Everything about it was wrong, despite on paper, we followed all the best practices. Obviously, 100 % test coverage is not actually a good metric. I know that. I did get that one changed to 70 at some point. I guess what's your broad takeaway from that circumstance?

1:17:56I think that one of the things that I've come back to over and over again in my career, and I think this first job really did teach me this, is as programmers, it's very easy to give up on our responsibility and say, I'm just doing what the business wants me to do and that's what my job is. My job is to do what I'm told, it's to complete this story. yes, I can maybe sometimes give input, but ultimately I make it happen, but I don't decide what we do. I'm the how, not the what. And I think that that's always been the problem I've seen at these companies when things went poorly was when developers just kind of gave up on doing what was good and what was right for the system and for their end users and for the code in order to just do what they're told.

1:18:50I think that as programmers, we have to accept the fact that we're not hired to do as we're told. Otherwise, we just wouldn't have the salaries we do. They wouldn't pay us this much just to not want our opinion. And even if they say they don't really want our opinion, we got to do what's right. If you're a good friend with somebody, you don't always do what they ask you to do, but you always do what's right for them. And that's how I think that we have to approach these things. And every company I've been at where the software made people's lives worse and not better, programmers had kind of given up on doing what was right and just did what they were told.

1:19:28Well said, Jimmy. Well said. Anything else, any stone that we've left unturned? I'm sure there'd be dragons elsewhere, but anything you'd like to highlight before we call it a show? No, I mean, yeah, there's tons of other stories. There's tons of other craziness, like the time I had to split the code base in half and all sorts of weird things like that. But they're a little long. So yeah, yeah, yeah. Fair. Well, great best worst blog post. Love that you wrote that up. I think there's a reason it was popular. And I think because it's a shared experience, well documented. And it's fun to laugh at these things.

1:20:06You know, we laugh so we won't cry. Or maybe we laugh at you and not with you because I didn't have that particular problem. But I appreciate you coming on the show and talking to us about it. and for giving us some big picture ideas and hope for the future of coding and also for the current state of coding and code itself. And a love for it that I share at least, even when I despise it sometimes. I still love it, and I think that that's just the way it is sometimes. Adam? I agree. I agree. All right. You never know when you're the best of times. That's right. Right? this seemed like from the outset like maybe not the best of times but actually it kind of was absolutely are you talking about our podcast or are you talking about his experience his experience his experience that was fun Jimmy thanks for sharing thanks for going I think deeper than maybe is necessary to share a story that may not even matter to anyone else besides you and you just shared it with the world and now you can reflect on some of those key attributes that really reflect back on a good time.

1:21:15Really. It's cool. Thanks. I enjoyed it. Yeah. All right. Bye y 'all.

1:21:22So Jimmy looks back on this time, this beautiful mess, as he said, the best worst code base as one of the greatest times in his life as a programmer. It's kind of funny how you can look back on times when you're in the moment and things kind of suck or they're not seemingly so awesome. And it's actually one of the best times of your life. It's kind of funny, right? As one wise person has told me before, these are the days. Okay, so that was a fun show with Jimmy. Very fun adventure. Very deep context we got to dive into. I hope you enjoyed the show today. Make sure you check out Jimmy's podcast, Future of Coding.

1:22:03you can find that at futureofcoding.org so I want to mention this here in the post show because we're still not quite there yet but we have definitely dove deep into the Zulip rabbit hole if that's even a thing I don't know maybe there's an analogy there I could have came up with but I didn't but we love Slack for many years and then recently we tried Zulip instead and so if you're already in Slack there's an invite there in the main channel you can take advantage of. To come over to Zulip and to test the waters, many of the folks who have done so already have said never go back to Slack, that Zulip is the best.

1:22:44And I think that's how we feel. We've talked about it on a recent episode of Change Login Friends, Jared and I. But I want to invite you. Go to the show notes. There's a link in the show notes for you to join our Zulip. Everyone is welcome. There are no imposters. Hope to see you there. We had some really awesome sponsors for today's episode. Assembly AI, the leader in speech AI. Check them out, assemblyai.com. Our friends over at Superbase, check them out at superbase.com. Our friends over at Speakeasy, check them out at speakeasy.com. and of course our friends over at test double test double.com just have test double build all your software and you'll be good there you go okay last but not least our friends our partners over at fly yes that's the home of changelaw.com and that is the public cloud built for developers who ship deploy your app in five minutes at fly.io okay those beats by break mess of cylinder with Banging.

1:23:56Gotta love BMC. That's it for today's show. We'll see you on Friday.