vo
The Changelog: Software Development, Open Source

Kaizen! Just do it (Friends)

20 Sep 2024 · 1 h 33 min
Listen in VO

Ask about this episode

Ask anything about it. ChatGPT or Claude reads this page and answers with the times it was said.

Ask ChatGPTAsk Claude

Connect VO and ask about every podcast you hear, including the moments you saved. Add to ChatGPT · Add to Claude

In short

Podcast Summary: The Changelog - Kaizen! Just do it (Friends)

Episode Overview In this episode of The Changelog, the hosts engage with Gerhard Lazu for Kaizen 16, discussing the development and implementation of new features and enhancements in their software projects. The conversation touches on topics such as speech AI models, the challenges of deploying applications, and the creation of custom feeds.

---

Key Discussions

  1. Introduction to Universal One
  2. Guest: Dylan Fox, founder and CEO of Assembly AI.
  3. Topic: Introduction of Universal One, a powerful speech AI model designed for high accuracy in speech recognition and understanding tasks.
  4. Features:
  5. Trained on 12.5 million hours of diverse multilingual voice data.
  6. Accessible through a user-friendly Playground GUI for testing before purchasing.
  1. Kaizen 16 Updates by Gerhard Lazu
  2. Gerhard presented updates and improvements made since the last Kaizen.
  3. Highlights included the introduction of a new repository for the Pipe Dream, a simple CDN built on Fly.io using Varnish Cache.
  1. The Pipe Dream Project
  2. Concept: Build a custom CDN focusing on simplicity and efficiency.
  3. Initial expectations were set around a simplified Varnish config, but the reality involved complexities and challenges.
  4. New developments included:
  5. Pull requests to enhance dynamic backends and cache management.
  6. Roadmap outlining future work, such as log management and purging strategies.
  1. Custom Feeds in Changelog
  2. Discussion on the new feature allowing Plus Plus members to create personalized custom feeds for podcasts.
  3. Feedback indicated strong user interest, with the first implementations showing success.
  4. Importance of user experience was emphasized, noting that such features should be intuitive and easy to use.
  1. Deployment Improvements
  2. The team discussed enhancements leading to faster deployment times.
  3. A switch to using Namespace for GitHub actions was highlighted, improving build and deployment speeds significantly.
  4. Ongoing efforts to optimize application boot times and deployment processes to reduce overall waiting times were discussed.
  1. Integration of Just CLI Tool
  2. Introduction of a new command-line tool called Just, designed to simplify local development workflows.
  3. Allows contributors to run scripts locally without needing to manage complex container setups.
  4. The tool aims to streamline development set-up and facilitate easier database management through commands.

---

Key Takeaways

  • Continuous Improvement: The episode showcases the ethos of Kaizen—continuous improvement in development processes and tools.
  • User-Centric Features: The emphasis on user experience when implementing new features, such as custom feeds, highlights the importance of catering to community needs.
  • Technical Enhancements: The integration of advanced CI/CD practices, like Namespace for GitHub actions and Just CLI, represent significant strides towards more efficient workflows.

Upcoming Topics

  • The conversation hinted at additional improvements and features that are in the pipeline, including further enhancements to the Pipe Dream project and the potential for broader usage of Neon database services.

---

Final Thoughts This episode of The Changelog exemplifies the ongoing journey of software development, focusing on collaborative efforts, innovative features, and responsive design to meet user needs. The hosts convey both challenges and achievements in their projects, reinforcing the importance of adaptability and user feedback in creating impactful software solutions.

Written by AI. May contain mistakes. Listen to the episode to check what was said.

Hear the part that matters, and keep it.Open this episode in VO. Double tap your headphones to save a moment as you listen.
Get VO free

Transcript

Automatic transcript. May contain errors.

0:14Welcome to Change Log and Friends, a weekly talk show about the perfect name. Thanks to our partners at Fly.io. Over 3 million apps have launched on Fly, including ours. You can too, in five minutes or less. Learn how at Fly.io. Okay, let's Kaizen.

0:40what's up friends i'm here with a new friend of ours over at assembly ai founder and ceo dylan fox dylan tell me about universal one this is the newest most powerful speech ai model to date you released this recently tell me more so universal one is our flagship industry leading model for speech to text and various other speech understanding tasks. So it's about a year long effort that really is the culmination of like the years that we've spent building infrastructure and tooling at Assembly to even train large scale speech AI models. It was trained on about 12 and a half million hours of voice data, multilingual, super wide range of domains and sources of audio data.

1:24So it's super robust model. We're seeing developers use it for extremely high accuracy, low cost, super fast speech to text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products. Very cool. So Dylan, one thing I love is this playground you have. You can go there, assemblyai.com slash playground, and you can just play around with all the things that is assembly. Is this the recommended path? Is this the try before you buy experience? What can people do? Yeah, so our Playground is a GUI experience over the API that's free.

2:01You can just go to it on our website, assemblyai.com slash playground. You can drop in an audio file, you can talk to the Playground. And it's a way to, in a no-code environment, interact with our models, interact with our API to see what our models and what our API can do without having to write any code. Then once you see what the models can do and you're ready to start building with the API, you can quickly transition to the API docs, start writing code, start integrating our SDKs into your code to start leveraging our models and all our tech via our SDKs instead. Okay. Constantly updated speech AI models at your fingertips.

2:36Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser. or you can get started with a$50 credit at assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI.

3:01Kaizen 16, Gerhard, what have you prepared for us this Kaizen? I think every time I don't know what to expect. And this time I do know what to expect. so what changed what's new what's fresh well i share the slideshow i mentioned last episode i have a slideshow but with my talking points couple of screenshots things like that this time i shared it ahead of time and i prepared ahead of time as well but also i've been making small updates to the discussion i think more regularly than i normally do discussion 520 on github i mean we always have one for every Kaizen. But this time I just, you know, went a little bit further with it and I think it will work well.

3:48Let's see. All right. We'll take us on this wild ride. Adam's also here. Adam. What's up? Hey Adam. Everything's up. Whenever someone asks me that, everything's up. That's the SRE answer. Everything's up. Everything is up. Otherwise I'm not here. If something's down, I'm not here. You know it's up because Gerhard's here. Yep. So everything's up i like that well last kaizen we talked towards the end about the pipe dream oh yeah that was the grand finale so maybe this time around we start with that we start with a pipe dream we start with what is new start where we left off exactly love it so we mentioned that or at least you mentioned jared that was it adam can't remember anyways we will clarify this after i mention what i have to say wouldn't be nice if we had a repository for the pipe dream self-contained separate from the application.

4:39Whose idea was it? I think it was both of ours. Adam said, can this be its own product or something? And I said, well, it could at least be its own repo. Something like that. That's right. So github.com forward slash the changelog forward slash pipedream is a thing. It even has a first PR that was adding dynamic backends. And we put it close to the origin, a couple of things so you can go and check it out, PR1. and uh what do you think about it is the repo what you thought it would be well for those who didn't listen to kaizen 15 can you tell us what the pipe dream is well i think the person whose idea it was should should should do that however however i can start so the idea of the pipe dream was to try and build our own cdn how we would do it single purpose single tenant running on fly.io it's running varnish cache the open source variant and we just needed like the simplest cdn that we needed which is i think less than 10 percent of what our current cdn provides and the rest is just most of the time in the way and it complicates things and it makes things a bit more difficult for the simple tasks how the idea started i would only quote you again jared Would you like me to quote you again?

6:04That was Kaizen 15. So many quotes. Sure, let's hear it. I like hearing what I have to say. I like the idea of having like this 20 line varnish config that we deploy around the world. And it's like, look at our CDN guys. It's so simple and it can do exactly what we want it to do and nothing more. But understand that that's a pipe dream. Right. That's where the name came from. Because the varnish config will be slightly longer than 20 lines and we'd run into all sorts of issues that we end up sinking all kinds of time into. Jared Santo, March 29th, 2024. Change along with friends, episode 38. Okay, so there you go.

6:43What's funny is, you know how when you're shopping for a car and you look at a specific car, maybe you buy a specific car and then you see that same car and color everywhere? After this, I have realized not just hearing the word pipe dream or maybe the words, if we can debate, is it two words or one? But I actually realized I say that a lot. I call lots of things pipe dreams, and I didn't realize it until you formalized it. And now I'm self-conscious about calling stuff pipe dreams. I think I did it on a show just the other day. I was like, dang it, because now it's a proper noun. And I feel like it's a reserved word.

7:16It's almost a product. Yeah, it's almost a product. If you could package up and sell 20 lines of varnish, we would do it. But if you can't, we would at least open source it and let the world look at what we did. So it has its own repo and it has its own pull request. So, you know, it's going to be a real boy. Does it work? Does it do stuff? I mean, I know you demoed it last time and it was doing things, but does it do more than it did before or is it the same? Yeah. I mean, the first, the initial commit of the repo was basically extracted what would have become a pull request to the changelog repo.

7:50That was initial commit and we ended up with 46 lines of varnish config. The pull request one, which added dynamic backends, and it does something interesting with a cache status header, we end up with 60 lines of varnish config. Why dynamic backends? That was an important one because whenever there's a new application deployment, you can't have static backends. The IP will change. Therefore, you need to use the DNS to resolve whatever the domain is pointing to. So that's what the first pull request was, and that's what we did in the second iteration now i captured what i think is a roadmap it's in the repo and i was going to ask you what do you think about the idea in terms of what's coming so the next step would be to add the feeds back end why because feeds we are publishing them to cloudflare r2 so we would need to you know proxy to that basically cache those i think that would be like a good next step.

8:55Then I'm thinking we should figure out how to send the logs to Honeycomb exactly the same as we currently send them. So that, you know, same structure, same dashboard, same query, same SLOs, everything that we have configured in Honeycomb would work exactly the same with the new logs from this new CDN. Then we need to do implement the purging across all instances. I think that's slightly harder because as we deploy the CDN in like 16 regions, 16 locations we would need to expire right like when there's an update so that i think is slightly harder but not crazy difficult and then we would need to import all the current edge redirects from our current cdn into the pipe dream and i think with that we could try running it in production i think good roadmap i dig it so our logs currently go to s3 not to honeycomb in terms of logs that we care about.

9:53And I know that I previously said we only care about our MP3 logs, not our feed logs in the sense of statistics and whatnot, but that has since changed. I am now downloading, parsing, and tracking feed requests like I am MP3 requests. And so we would either have to pull that back out of Honeycomb, which maybe that's the answer, or somehow have it also write to where s3 is currently writing to in the current format for us to not have major rewriting on the app side thoughts on that so we can still keep s3 whatever intercepts the logs right because in our current cdn obviously the cd intercepts all the logs and then some of those logs they get sent to s3 indeed but then all the logs that get sent to honeycomb so you're right, I forgot about the S3 part.

10:48So on top of sending everything to Honeycomb, we would also need to send a subset to S3 exactly as the current config. So yes, that's an extra item that's missing on that roadmap, indeed. All right, cool. So we add that item to the roadmap, and I think it's all honky dory. Do you know how you're going to implement purge across all app instances? Like, what's the strategy for that? No idea. No idea currently. I mean, based on our architecture and what we have running so that we avoid introducing something new as a new component, a new service that does this, we could potentially do it as a job using GoBan, I think.

11:32Because at the end of the day, it's just hitting some endpoints, HTTP endpoints, and it just needs to present a key, right? If we don't use it, anyone can expire our cache, which is a default in some CDNs I know. Yeah, we found that out the hard way. Exactly. So that's something that we need. I think an O-band job would make most sense. It's actually pretty straightforward. We already have a FastlyPurge function in our app that goes and does a thing and then we just change this to go and background Java reset on all these different... Now, there has to be some sort of orchestration of like the instances have to be known.

12:08Maybe that's just like a call to fly or something. or I don't know how. DNS. Okay, DNS based, yeah. We can get that information by doing a DNS query and it tells us all instances and then, yeah, we can get all the URLs. Yeah, that sounds like a straightforward way of doing it. Where's the data being stored when we upload? Currently? In Pipedream. Pipedream is just a cache, so. You mean where's the cache data being stored? Okay, so Pipedream is just, what exactly does Pipedream do? So Pipedream is our own CDN, which caches requests going to backends. So imagine that there's a request that needs to hit the app and then the app needs to respond.

12:52So the first time, like let's say the homepage, right? Once the app does that, subsequent requests, they no longer need to go to the app. Pipedream can just serve because it already has that request cached. and then because PipeDream is distributed across the whole world, it can serve from the closest location to the user. Exactly. And same would be true, for example, for feeds, even though they are stored in Cloudflare R2, the PipeDream instance now goes to Cloudflare R2, gets the feed and then serves the feed. Gotcha. And so Varnish is storing that cache locally on each instance in its local disk storage or however Varnish does what it does.

13:32so by default we're using memory but using the static back end like a disk back end would be possible yes i was just thinking about expiring because we just did this yesterday where we had to correct deployed slash published episode and we ran into a scenario where fastly was caching obviously because it's the cdn and then i went into the fastly service and purged that url And then it wasn't doing what we expected. And I bailed on it and handed it to Jared. And Jared checked into R2 and R2 was also caching. And so we essentially had this scenario where our application was not telling the CDN that this content is new, expire the old, purge, etc.

14:16And I just wonder, in most cases, aside from the application generating new feeds, which happens usually at the action of a user, so me, Jared, somebody else publishes an episode or republishes, couldn't the expiry command, so to speak, come from that action and inform the CDN? Yeah, exactly. Which is how it works right now with Fastly. Like after you edit an episode, we tell Fastly to purge that episode. The problem we had yesterday is that Fastly purged it, but then Cloudflare also had a small cache on it. And so Fastly would go get the old version again and say, okay, now I'm fresh. And so we had two layers of cache that we didn't realize.

15:01and so that's probably fixed now but yes it would be basically everywhere in our app that we call fastly.purge we would just replace that with pipedream.purge or whatever which would be an oban process that goes out to all the app instances i see so the question was mechanically how to actually purge the cache not so much when yeah because we already have when pretty much figured out gotcha which is pretty straightforward really because it's when we publish and we edit or delete like those are the times that you purge the cash otherwise what's the point yeah like otherwise you don't do it please don't make any sense now you're just change hasn't happened so don't change so okay how plausible is this pipe dream like is it should we rename it to like something else because it's not a pipe dream anymore or more or less of a pipe dream obviously i'm not suggesting that naturally but like it becomes real does it become an oxymoron when it becomes real.

15:59I don't know. I quite like the name, to be honest. I think it has a great story behind it, you know? So it just goes back to the origin. And the CDN is a pipe, right? I mean, it is a pipe. Yeah, exactly. Yeah. Yeah, I like that pipe idea. That was like one of the follow-up questions. Do we keep a space or introduce a space or no space? That's a really important decision. Space or no space? What about a tab? Should we put a tab in there? We can. Camel case, no space, space. what do what the listeners think i mean you've you've been hearing this story for a while and you've you've heard us thing i think we should have a poll and that's how you know i know that's that that's how we end with names like boaty mcbotez we're very aware of that no this is not that we're just asking like how do we how what would be the the way to spell it that would make most sense pipe dream one word pipe space dream pipe tap dream i'm not sure about that i think we can do one like us for fun or camel case indeed i'm leaning towards one word the merriam-webster dictionary and the cambridge dictionary both say that it's two words i'm seeing it two words everywhere yeah except for old english yeah where it was pip dream all one word i'm leaning towards one word though just like okay just pip dream one word okay and i'm leaning in the other direction so we need a poll great well the repo name is already like lowercase pipedream no spaces no nothing no no dashes nothing like that so you know i think it would make sense so yeah all right we'll run a poll see what people think see what people want give the people what they want correct and when it comes to when we do switch it into production whenever you know that that happens I think we could maybe discuss again, whether we rename it, when it stops being a Pipe Dream for real.

17:50For now, it's still like a repo. It's still a config. It runs. I mean, if you go to pipedream.changelog.com, you know, it does its thing, but it's not fully hooked up with everything else that we need. I have a new name. Pipe Reality. Pipe Reality. Just let it marinate. Not now. Not yet. Pipe Media. Pipe Media. I don't know. Pipe log? Pipe log. Ooh. Oh, here's a better one. Change pipe. Pipely. Pipely. Ooh. That one really hurts. I think that's the winner. I think that's the winner. Oh, goodness. Quick, buy the debate before someone else buys it. Pipe.ly. Oh, yes. That one's almost too good. Almost.

18:37Yeah. Is this really where we're marching towards? I know this began as literally a pipe dream, and it's becoming more real. You've had some sessions. You've, according to the, maybe I'm jumping the gun a little bit on your presentation here, but you've podcasted about this slash live demoed this. We've been talking about the name. We've been talking about the roadmap. Is this really a true possibility to do this successfully? Well, based on the journey so far, I would say yes. I mean, it would definitely put us in control of the CDN too. a CDN is really important for us so it's even more important than a database because we're not heavy database users and we'll get to that in this episode I'm sure so a CDN really is the bread and butter now we need something really simple we need something that we understand inside out we need something that I would say is part of our DNA because we're tech focused and we have some great partnerships and we've been on this journey for a while you know it's not something that one day we woke up and we said let's do this so this has been in the making for a while we're almost forced in a way yes i would say encouraged you know in a way like we're pushed in this direction there are other options yeah but i think there is like this like natural progression towards this and it doesn't mean that we'll see it all the way through but i would say that we are well on our way to the point that i can almost see the finish line i mean even the roadmap right putting the roadmap down on paper it made me realize actually the steps aren't that big and we could take them comfortably between kaizens and i don't want to say by christmas but wouldn't it be a nice gift a Christmas gift.

20:23What do you think? I think that's a bold roadmap. Let me add this to the roadmap or maybe I'm not seeing it in the repo and it's there. Test harness. Is there a test harness? No, there isn't a test harness now. I would love to be able to develop against this with confidence, especially once we start adding those edge reader racks and different things. I would love to have that as part of the roadmap so that I can fire it up and create an issue. I would love that. Yeah, go for it. Cool. Open source for the win. Cool. So I'm going to open source the issue and then you open source the code. Amazing.

20:58I love that. Just making sure you didn't say PR is welcome and you're moving on. Cool. Yeah. Can we revisit the idea of this being a product? Single tenant, single purpose, simple seems like a replicated problem set. Honestly, I think so. Honestly, I can definitely see this being part of Flutter.io. Well, there's this name for which we cannot name in regards to Fly. It's more of a class of people, I would say, is probably that. I'll be even more vague. Sorry, listeners. That's so vague that I don't even know what you're talking about. There is some information. I'm not sure how much we can share.

21:36But then there's like Tigris that has led the way in a lot of ways. And I just talked to Oves because, by the way, they may even be sponsoring this episode. Fly is not only a partner, but also a sponsor of our content. and I had a conversation with Oves, who is one of the co-founders of Tigris, and he shared with me that if it weren't for Fly, it would have taken them years to build out all of the literal machines across the world with the NVMe drives necessary to be as fast, to be what Tigris has promised. And I don't want to spoil it for everybody, but Tigris basically is an up-and-coming S3.

22:13and because of the way that Fly networks and because the way that Fly handles machines across the world and the entire platform that Fly is very developer-focused, Tigris was able, I think within nine months, to stand up Tigris. And so you can deploy Tigris via a single command in the Fly CLI and then you can also have all your billing handle inside there. This is not an ad. I'm just describing it. but you know when I said that back in the day I was thinking about tigers because I first learned about them and knew about this story and I knew they were built on fly I knew their story was only possible because of what fly has done and I think that this pipe dream is realized or capable being realized because of fly being what fly is and I feel like we have this simple nature sort of the I said really simple cdn but I'm not tied to that because rss is you know kind of one the really simple part of it.

23:13But I think that's kind of what it is. I feel like other people will have this and it can certainly live in this world of fly. I don't know. There's a possibility there. I think we build it for ourselves and then we'll know more. Are you thinking, make it private? The repo? It's still not too late. Are you going to rug pull these people before there's a rug down? Well, yeah, no one's using it. Yeah, private rug. and we're at 60 lines of varnish i think we're getting ahead of ourselves right i think so but once we start adding the test harness once we start adding the purging which by the way is specific to our app but maybe that would need to be generic by the way so if we this was to be a product we would need to have a generic way of purging doesn't matter what your app is so there's a couple of things that we need to implement to make this as a product and in that case it would be in this repo i think but um it could also be like a hosted service like tigris is maybe especially if we get the cool domain why not i can see that and this can be our playground like the pipe dream can be our playground but then the real thing with all the bells and whistles could be private yeah i think we build pipe dream in the open and then if we aside there's a there's a possibility there then you genericize it yeah in a separate effort the one thing which i do want to mention is that there's a few people that helped contribute so i'd like to this is also time for shout outs of course to matt johnson uh one of our listeners shout out to matt and also james a rosen he was there from the beginning so the first recording that we did that's already live the second one as well that we recorded i haven't published it yet i still have to edit it but that was like basically the second pull request that we got together and even though a bunch of work you know went obviously in the background before we got together when we did get together was basically putting all the pieces you know so we did like in this very open source group spirit and um yeah so there's that so i think keeping that true to open source would be important and if not then we would need to you know make the decision soon enough so we know which direction to take but you're right rug pulls not a fan at all we should never do that and even the fact that we're discussing so openly about this i welcome that i think it's amazing there's transparency so that we're always straight from the beginning what we're thinking so that no one feels that they were misled in any way agreed agreed like it well the last thing i would like to mention on this topic before i'll be ready to move on is that we live stream the CDN Journey, a changelog with Peter Mbannugo.

26:02There'll be a link in the show notes. We got together and we talked about where we started, you know, how we got to the idea of the pipe dream and where we think of going. So if you haven't watched that yet, it'd be worth... There was a slideshow. Not as good as the last one, the last Kaizen, but it was... I'm happy with it. Let me put it that way. Awesome. Cool. We'll link that up.

26:34Okay, friends, here are the top 10 launches from Superbase's launch week number 12. Read all the details about this launch at superbase.com slash launch week. Okay, here we go. Number 10, Snaplet is now open source. The company Snaplet is shutting down, but their source code is open. They're releasing three tools under the MIT license for copying data, seeding databases, and taking database snapshots. Number nine, you can use PG Replicate to copy data, full table copies, and CDC from Postgres to any other data system. Today, it supports BigQuery, DuckDB, and MotherDuck with more syncs to be added in the future.

27:16Number eight, Vect2PG, a new CLI utility for migrating data for vector databases to Superbase or any Postgres instance with PG Vector. You can use it today with Pinecone and Qdrant. More will be added in the future. Number seven, the official Superbase extension for VS Code and GitHub Copilot is here. And it's here to make your development with Superbase and VS Code even more delightful. Number six, official Python support is here. As Superbase has grown, the AI and ML community have just blown up Superbase, and many of these folks are Pythonistas. So Python support expands. Number five, they released log drains so you can export logs generated by your Superbase products to external destinations like Datadog or custom endpoints.

28:04Number four, authorization for real-time broadcast and presence is now public beta. You can now convert a real-time channel into an authorized channel using RLS policies in two steps. Number three, bring your own Auth0, Cognito, or Firebase. This is actually a few different announcements, support for third-party auth providers, phone-based multi-factor authentication, that's SMS and WhatsApp, and new auth hooks for SMS and email. Number two, build Postgres wrappers with Wasm. They released support for WASM WebAssembly foreign data wrapper. With this feature, anyone can create an FDW and share it with the Zootbase community.

28:48You can build Postgres interfaces to anything on the internet. And number one, Postgres.new. Yes, Postgres.new is an in-browser Postgres with an AI interface. with Postgres.new you can instantly spin up an unlimited number of Postgres databases that run directly in your browser and soon deploy them to S3. Okay, one more thing. There is now an entire book written about Superbase. David Lorenz spent a year working on this book and it's awesome. Level up your Superbase skills and support David and purchase the book. Links are in the show notes. That's it. Superbase launch week number 12 was massive.

29:34So much to cover. I hope you enjoyed it. Go to superbase.com slash launch week to get all the details on this launch. Or go to superbase.com slash changelogpod for one month of Superbase Pro for free. That's S-U-P-A-B-A-S-E dot com slash changelogpod. What's next? Custom feeds. That's one of your topics, Jared. Custom feeds. So tell me about it. I don't know what it is. I know what it is, but I don't know what exactly about custom feeds you wanted to dig into. So custom feeds is a feature of changelog.com that we wanted to build for a long time. Probably not quite as long as we waited on chapters, but we've been waiting.

30:21Mostly because I had a false assumption or maybe a more complicated idea in mind. We wanted to allow our Plus Plus members to build their own feeds for a long time. The main reason we want to allow this is because we advertise ChangeLog Plus Plus as being better. Don't we, Adam? Yeah, it is better. It's supposed to be better. However, people that sign up and maybe only listen to one or two of our shows, whereas they previously would subscribe publicly to JS Party, for instance, and maybe ship it, they now have to get the plus plus feed which was because of supercast all of our episodes in one ad free master feed and so for some people that was a downgrade because they're like wait a second i want the plus plus versions but i also don't want all your other shows to which we were quite offended but we understand and that's been the number one request i would i would call it a complaint, but actually our supporters have been very gracious with us.

31:25They ask for it, but they don't, they say it's not a big deal, but it would be nice. In fact, some people sign up for plus plus and continue to consume the public feeds because that's what they want to do. We wanted to provide a solution for that for a very long time. And because it was plus plus only, I had it in terms of like blockers. I had this big blocker in front of it, which was, we need to get off Supercast first because that's the reason why it's a problem is because Supercast works this way which is our membership system that's built all for podcasters and it's served us very well but it has some technical limitations such as this one so moving off Supercast is a big lift and not one that I have made the jump yet because there's just other things to do and it works pretty well and lots of reasons and so I didn't do custom feeds for a long time thinking well we got to get off of Supercast first.

32:15And then one day it hit me. Why? Why do we have to get off of Supercast? Can't we limp into this somehow? Can't we just find out a way of doing it without getting off of Supercast? And the answer is actually pretty simple. It's like, well, all we need to know is, are you a Plus Plus member or not locally to our system, which lives in Supercast? And then I remembered, well, Supercast is just using Stripe on the backend and it's our Stripe account. And that's awesome, by the way, they give us direct access to our people and no lock in and stuff. And so kudos to them for that. And so I was like, no, all we actually have to know is, do you have a membership?

32:49And all the membership data is over in Stripe. And so it's simply a Stripe integration away from having membership information here in changelog.com. So I built that, worked just fine. And then I realized, okay, now I can just build custom feeds and just allow it to people who are members. And so we build out custom feeds and it's pretty cool. Have you used them, Gerhard? Have you built a custom feed? No, I still consume the master feed, the master++ feed with everything. Master++ feed. Yeah. Okay, that's fair. But do you know what I would love to do? To build one now. Oh, you would? Yeah. Live on the air.

33:22Let's see what happens if we do that. So changelog.com, how do I do that? Like run me through that, Jared. I sign in. Are you a++ member? Yeah, of course you are because you have the++ feed. Yeah. Okay, so sign in to changelog.com. Yep. And go to your home directory. the tilde yes i'm there and there you should see a section that says custom feeds i do see it okay click on that sucker get started okay new feed all right there you go add a feed now you're going to give it a name that's required you know call it gerhard's feed yes sure gerhard's you can write your own tagline and that'll show up in your podcast app okay you can be like it's better hang Hang on.

Read the full transcript

34:05I'm still at Tankline. Jared made me do this. Okay. Okay, moving on. Then you get to pick your own cover art because, hey, maybe you're making a single show feed. Maybe you're making all the shows. You can pick the plus plus one. You can pick a single show. Pick your cover art or you can upload your own file. You get to pick a title format. So this is how the actual episode titles come in to your podcast app. So maybe you want to say like the podcast name, colon, the title of the episode. Maybe you just want episode titles. You know, put a format in there. And then you can limit your feed to start on a specific date.

34:44Some people want like fresh cuts between their, like the old days and the new days. And so they want to start it on this date because it doesn't mess up there marked as red or whatever. September 13th, start today. It'll start today. Okay. It's going to be empty. And then pick the podcast you want. Okay. so hang on i used oh i see okay okay i see i see so upload cover art that's the thing which was messing with me because i wanted to add mine but then it said or use ours and when you say or use ours i'm basically changing the cover art which i uploaded with one of yours interesting right ours as in a changelog cover art that previously exists got it so you can like use js parties or upload your own file and you'll have your own cover art for your own feed.

35:38Okay, so I've made a few changes. First of all, the name is Gerhard and Friends. Okay. Description is Kaizen 16, this episode. Okay. The cover art, I uploaded one, but then I changed it to Changelog and Friends. Okay. Starts today, 13th of September. Yes. Title format, I will leave it empty. And for the podcast, I'll choose Changelog and Friends. Okay. Yes. and this feed should contain change of plus plus at free extended audio yes bam and automatically add new podcasts we launch i'm going to deselect that because i only want change your friends save perfect boom it's there there you go you build a custom feed you can grab that url pop it into your podcast app subscribe to it got it and i found the first bug no you didn't so the bug is if I upload my cover art and then I select another cover art from one of yours, it uses my cover art, but not in the admin.

36:40In the admin, it shows me that it's using yours, but when I create the feed, it's using my cover art. Okay. So you did both. I did both. Yes. And then submitted the form? Correct. Yes. Okay. You are the first person that's done that, I think. Of course. Of course. People usually pick one or the other. Yeah. So, okay. Open an issue. I will get that fixed. I will. Let me take a screenshot so that I remember. Boom. There. Awesome. Cool. Looks great. Actually, hang on. The picture which I want for this cover art is us three recording right now. So if Adam looks up, I'll take a screenshot. There you go.

37:16That will be my cover art. Okay. Got it. So good. Too good. So custom. So cool. You know one thing I was going to do, which I haven't done yet, this is a reminder is I want to put the changelog legacy cover art in the list. Don't you think so, Adam? Like you can have the old changelog legacy logo if you want. That would be cool, actually. Yeah. Super dope. Actually, that's an idea we had is like to expand these, you know, to like a bunch of maybe have like custom artists come in and create new cover art you can select from. That might be cool. Very cool. But yeah, it's been kind of a screaming success, honestly.

37:49We have currently we have 320 changelog++ members. and those 320 people have created 144 custom feeds so far. Including mine. Including yours. I see yours right there. Amazing. And the cover is your face. Correct. Yes. Cool. So cool. So that's the feature. That's amazing. It worked very well, I have to say. I just still have to load it in my podcast player, but once I do that, amazing. Well, let's stop there then because that's where I'm at and that's where I'm stuck, Jared. You're also stuck? Yes. so Gerhard's next step is to do what I've done and I think he may have the same outcome I don't know my outcome was I loaded the URL into my clipboard on my iPhone opened up Overcast add podcast via URL did that, clicked add URL and it says not a valid URL does yours have a start date?

38:50no Okay. I don't think so. So yours has a bunch. The URL only contains feeds. It's forward slash feeds, forward slash Asha. So it doesn't have the full. Oh, I might have screwed that up yesterday when I was fixing something else, when I was giving you your comment. Well, this has been weeks for me. I just haven't reported it to you yet. And you've been waiting for this to do it live? Why would you wait this long? Are you waiting for this? Yes. Public embarrassment. Okay. No, just the fact that I just haven't done it yet. I'm sorry. Okay. No, that's all right. I think that that copy URL button should have copied the entire URL.

39:24Did it just give you the path, Gerhard? It did, yes. No wonder it's not a valid feed. So I literally fussed with that yesterday because I was giving Adam a different copy paste button and I might have broken it yesterday. Now, interestingly, if I hover over it, I can see the correct link. Yeah. But when I click on it, I only get the path. Yeah, the href is correct, but the data dash copy value is incorrect. Right. And I'm pretty sure I broke that yesterday. So that used to work because all these other people are happy. But you're sad because I broke it yesterday. So I have a quick fix. You right click the get URL.

39:59Yeah. And you say copy URL rather than relying on the click action. Right. And then you get the proper URL. Try that, Adam. Let's see if that works. Let's see here. Copy link. Did it solve my problem? Let me enter it. Boom goes the dynamite. It's at least not yelling at me. it is taking its time though well the other reason why that was happening probably a few weeks ago is because if you load a feed that has all of our episodes for instance a thousand plus 12 megabyte xml file we would serve it slow enough that overcast would time out and it wouldn't think it was a valid feed but then I fixed that by pushing everything through the CDN because at first when I first rolled it out it was just loading directly off the app servers and it was just a little bit too slow for overcast Right.

40:46Okay. Next question then. This is a UX question. I am not a plus plus subscriber, but I can click the option and I assume it does nothing to say this feed should contain plus plus ad for extended audio. I haven't clicked play because I just literally loaded it for the first time now, but I'm assuming that I won't have plus plus content because I'm not a plus plus subscriber. Is that true? No, I do have plus plus content. I'm thinking you are an admin and so it doesn't matter. Okay, gotcha. So does this check then only show up for people who can check it? The entire UI for building custom feeds only shows up if you are an active plus plus member or an admin, which is literally the three of us.

41:35Okay, that makes more sense then. Like you can't even build custom feeds. Now I did consider custom feeds for all. you know let the people have the custom feeds but plus plus people obviously would only get be the only ones who get the checkbox that's something that i'd be open to if lots of people want it but for now i was like well let's let our plus plus people be special for a while is there a cost center with these custom feeds like is there an additive to the cost if we were having to deal with costs marginal okay every custom feed has to be updated every time an episode's updated and so if we had 100 ,000 of them there would be some processing and maybe hit some R2 too many put actions versus, you know, it's free egress but it's not free all operations.

42:22And so there's like class A operations, class B operations and the more you edit those files and change them I think eventually those operations add up to costing you money but it's marginal on the margins. If it got to be a huge feature where I mean if we had 100 ,000 people doing custom feeds, we'd find a way of paying for that. You know? Yeah, it's a different problem. But yeah, it's a marginal cost, but not worth considering. Gotcha. Okay. So the copy can be updated pretty easily. It's probably a fix going on already for that because it's so simple. For the ships, it'll be out there. Good.

42:56Well, because I mean, I was like, well, how do I get this URL to my iPhone? I guess I can like copy it and like airdrop it to my iPhone. Maybe it'll open up in the browser. and I was like, well, let me just go on the web and get URL, essentially. Yes, our user experience assumes that our users are nerds. And so far, before I broke that copy button yesterday, there's been zero people who are like, now how do I get this into my podcast app? No one's asked us that because all of our Plus Plus members completely understand how to copy and get into their whatever. They are smarter than me, most of them.

43:30Now, if it was for a broader audience, if this was a baking show, and we're going to provide custom feeds for bakers or aspiring bakers, then I probably would have to add more of a handholding. And Supercast actually does a really good job of handholding you into your private feed because it's not a straightforward mental process for most people, just for nerds. Yeah. Yeah, I agree. It kind of requires some workaround. There's really nothing you can do about that, right? I mean, you're adding literally a custom feed via URL that no index knows about. So it's obvious you have to do some sort of...

44:05workaround to get there, to get your feed into your... Yeah, I mean, a better UX would be after the custom feed's created, we send you an email. That email contains a bunch of buttons. Each button's like add to Overcast, add to Pocket Cast, add to Apple Podcasts. And depending on, you know, now I'm just... I like that idea a lot. That's how Supercast works. Yeah, I like that idea a lot. Email them every time it changes, that they go upon creation. And now that is immutable until, well, theoretically mutable until they edit it again and then it's muted you know so it's it's in stone yeah it's mutated we could certainly add a button that says email this to me you know next to the get url maybe like email me the url it's a good idea and that's like a fast way to get it into your phone without having to do phone copy paste or airdrop like gerhard did yeah because you don't know about the email happening so that's a good feature even for nerds because it's just easier that way.

44:57Well, that would have solved the problem of me having to get the data onto my iPhone. Totally. Which my email is. Exactly. I think we should add that as a feature. It's a good idea. Hey, Jared here in post. That email it to me feature just shipped today and that copy paste bug fixed. Kaizen. Custom feeds are here, y 'all. If you're a Plus Plus subscriber, by the way, changelog.com slash Plus Plus. It's better. If you are not a Plus Plus subscriber and you desperately want this feature, let us know. Because, you know, squeaky wheels and oil. Must be in Zulip. I don't know. Is the other catch, right?

45:40Anyways. Well, not even, Gerard's not even Zulip yet, so let's not get ahead of ourselves. No, but what's the URL? Because I would like to join. Changelog.zulipchat.com. Okay. But can you just get on from there? I don't know. It's new to us. Zulipchat.com. I'm doing it now. Let's see. Log in. Okay. Log in to Google. Go. There you go. Yes. Continue. Okay. Sign up. You need an invitation to join this organization. All right. Go to our Slack. Go to main. Scroll up a little bit. You'll see there's an invite link. To get into Zulip, you have to go to Slack. It's a Trojan horse. That's how you do it.

46:20That's right. You install one through the other. Listeners, you can do this too. You can follow the same instructions. It is in main. I think it's Friday, September 6th. Jared posted it as a reply to after that conversation. Now we're trying out Zulip in earnest. And there's a link that says joiners will appear. And it's a long link that I could read on the air, but no one would ever hand type that in. No. Look at that. I agree. Can't put it in the show notes though. So it might be there. So there you go. Yeah. We've shared our thoughts already elsewhere on friends with this, but you know, I'll be, I'd be curious.

46:53We'll be so many Kaizens away. well at least one more Kaizen away multiple months before we get Gerhard's by the next Kaizen we may be like transitioned over to Zulip we might be self-hosting it but I don't think we should do that no way there's a Kaizen channel this makes me so happy and it's for all ideas about making things better and stuff I even put one in there you can read it oh wow okay I'm definitely going to check this out this is nice this is a very nice surprise it was worth joining just for this oh wow so cool this is nice yeah isn't that cool I thought a Kaizen channel would be on point so cool so I was kind of thinking like well how do we replicate our dev channel over here and it's like well dev is just one thing like let's have a kaizen and then different topics can be based on thumbs up yeah big thumbs up so amazing all right awesome custom feeds zulip chat kaizenine what's next on the docket well i'm going to talk about one very quick improvement actually two which i've noticed the news yes i love the latest oh you like that That graphic is so cool.

47:51I really like the small tweaks. Also, the separators, the dividers between the various news items. They just stand out more. I really, really like it. And it feels like more, the play button is amazing, by the way. I love it. I can definitely see it. I mean, the play button stand out. Yeah. It feels so polished. Thank you. It really does. But the latest is so amazing. And the news archive, it's there. And it works. Yes, it is. Amazing. I appreciate your enthusiasm and to tell everybody what the latest is, I literally put an arrow and the words of the latest on our homepage that points to the issue because it's kind of, it could be discombobulating.

48:30Like you look at it on a desktop, at least. On mobile, it goes vertical. But like on the left is kind of the information about news and the signups and stuff. And on the right is the latest issue. But you may not know, like, what am I looking at when you land on the page? What's the thing on the right-hand side? And so I just put this little arrow, handcrafted with SVG, by the way. And the words, the latest, like someone just scratched them on the page that points to that issue. So it's just kind of giving you a little bit of context and Gearheart loves it. So I appreciate it. Gives it another dimension.

49:02It's playful. It's, you know, like there is some fun to be had here. It's not just all serious. It's not like another news channel, but it's really, really nice. Like the whole thing, it feels so much more polished compared to last time. I can definitely see like the tiny, tiny improvements. Very cool. So much Kaizen. Indeed. Polishing. Cool. Well, the next big item on my list is to talk about twice 2x faster time to deploy. This is something we just spent a bit of time on. I was surprised by the way of the latest deploy. It was slower than 2x, but we can get there. Okay. The first thing which I would like to ask is how do you feel about our application deploys in general?

49:48Like, does it feel slow? Does it feel fast? Does it feel okay? Do you feel surprised by something? How do application deploys when you push a change to a repo feel to you? Historically or after this change? Historically. Historically, I would say too slow. Too slow. Okay. Adam? Yeah, historically too slow. Okay. Okay. So what would make them not too slow? Like, is there like, um, maybe like a two X duration? That's so leading though. I didn't be like, I literally meant like how many minutes or seconds I think you talked about that. Would it feel that it's, it's good enough? There's like this threshold that I'm not sure exactly.

50:31It's probably fuzzy, but it's the point where like you're waiting so long that you forget that you're waiting and you go do something else. and I think that's measured in single digit minutes but not necessarily seconds. Like I can wait 60 seconds. Well, that's my seconds. I can wait one minute and maybe I'm just hanging out in chat waiting for that thing to show me that it's live. But as soon as it's longer than that, I'm thinking, well, I should come back in five. Then I forget what I was doing. I don't come back and I've lost flow basically. So I would say around a minute, 30 seconds would be spectacular.

51:09It doesn't have to be instant, but I think two, three, four, five minutes, it's getting to be where you're like, yeah. It's kind of like friction to deploy because you deploy and you're like, now I got to wait five or 10 minutes. That's my very fuzzy answer. Okay, that's a good one. So what used to happen before this change, we used to run a Dagger engine on the fly so that it would cache previous operations so that subsequent runs will be much quicker especially when nothing changes or very little changes the problem with that approach was that from github actions you had to open a wireguard tunnel into fly so that you'd have that connectivity to the engine and what would happen quite often is that tunnel for whatever reason would maybe be established but you couldn't connect to the instance correctly and you would only find that out a minute or two within the run and then what used to happen you would fall back to github which is much slower because there's no caching there's no previous state and the runners themselves because they're free they are slower two cpus and seven gig which means that you have to when you have to recompile the application from scratch it can easily take seven eight ten minutes and that's what would lead to those really slow deploys so what we did between the kaizens since the last kaizen let me see which pull request was that it was pull request 522 so you can go and check it out to see what that looks like so when everything would work perfectly when the operations would be cached you could get a deploy within four minutes between four and five minutes thereabouts and with this change what i was aiming for is to do two minutes or less and when i captured when i ran this like the initial tests and so on so forth we could see that while the first deploy would be slightly slower because you know there was nothing subsequent deploys would take about two minutes two minutes and 15 seconds the one which i have right here which is a screenshot on that pull request 522 so how did we accomplish this we're using namespace.so which they provide faster github actions runners basically faster builds and we run the engine there and when a run starts we basically resume restore everything from cache the namespace cache which is much much faster and we can see up there basically like per run we can see how much cpu is being used we can see how much memory again And these are all screenshots on the pull request.

53:52And while the first run obviously used quite a bit of CPU because you have to compile all the Elixir into bytecode and all of that, subsequent runs are much, much quicker. And the other thing which I did, I split the, let's see, is it here? It's not actually here. We need to go to Honeycomb to see that. So I'm going to Honeycomb just to look at that. I've split the build time, basically the build, test, and publish from the deploy time because something really interesting is happening there. so let's take for example before this change let's take dagger on fly one of the blue ones and have a look at the trace so we have the this previous run which actually took four minutes and 21 seconds and all of it is like all together it took basically three minutes there's like some time to start the engine to start the machine whatever whatever all in all four minutes and 20 seconds so a newer run for example this one which was fairly fast it was two minutes and a half if we look at the trace we can see that diagram namespace the build test and publish was 54 seconds so in 54 seconds we went from just getting the code to getting the final artifact which is the container image that we ship into production in this case we basically publish it to ghcr.io.

55:10And then the deploy starts. And the deploy took one minute and 16 seconds. So we can see that, you know, like with this split is very clear where the time is spent. And while the build time and the publish time is fairly fast, I mean, less than a minute in this case, the deploy takes a while because we do blue green and new machines are being promoted. The application has to start. It has to do the health checks. So there's quite a few things which happen behind the scenes that, you know, if you look at it as like one unit is difficult to understand. so this was ideal case this is what i thought would happen of course the last deploys if i'm just going to filter these dagger on namespace by the way we are in honeycomb we send all the traces and all the build traces from github actions to honeycomb and you can see how we do that integration our repo you can see that we had this one 2.77 minutes right which is roughly 240 but the next one was surprising which took nearly five minutes and if i look at this trace this was again nothing changed like stuff had to be recompiled but in this case the build test and publish took nearly three minutes which this tells me there's a there's some variability into the various runs when it builds it i don't know why this happens but i would like to follow up on that as a tldr this change meant that we have less moving parts and when namespace works and this is something again that we need to understand why did this run take longer it should take within two minutes we should be out like a change should be out in production half the time is spent in build and half the time is spent on deploys so when it comes to optimizing something now we get to choose which side do we optimize and i think build test and publish is fairly fast the slower part is the actual deployment so how can we maybe half that how can we get those changes once they're finished and everything is bundled how could we get it out quicker i love it i think do you have ideas on that well i think the application boot time could be improved right because it takes a while for the app to boot when i say it takes a while it may take 20 30 seconds for it to be healthy all the connections to be established now i'm not sure exactly which parts of those you know would be the easiest one to optimize but i think the application going from the deploy starting and the deploy finishing taking a minute and a half is a bit long so i'll need to dig deeper like is it when it comes to connecting to the database is it just the application itself being healthy like which part needs to be optimized but again we're talking this is like a minute and a half we're optimizing a minute and a half just to put this into perspective.

57:54And that's why I started with the question, like how fast is fast enough? Yeah. I mean, I think if you're at 90 seconds, you're probably right about there. I would still go in and spend an hour thinking like, is there a low hanging fruit that we haven't looked at yet that we could squeeze 10 more seconds off? And then I would stop squeezing the radish after that. I see. That'd be my take on it, Adam. Well, the flow, it seems, is every time new code is pushed to our primary branch on the repository, a new deploy is queued up. And this process happens for each new commit. to the primary branch. A new application is spun up, it's promoted, so if I deploy slash push new code, and then a minute later, Jared does the same thing, my push does this process, my application is promoted, Jared's commit does the same thing, his application is then promoted.

58:54And that's via networking, and then these old machines are just, you know, thrown off, and then the new machines are promoted, and they just fall by the wayside. Correct. Which totally makes sense. I think you have things happening that we want to happen. I agree with you on the low-hanging fruit, but on the app boot process, we've got even things like 1Password being, those things being injected from their CLI. I'd imagine that API call is not strenuous, but it's probably seconds, right? so there's probably in each thing we're booting up as part of the app boot process for every commit there's at least one to several seconds per thing we're instantiating upon boot well that's just me hypothesizing how things work because no that's that's a good one that's that's exactly what you know we're like trying to hash it out so that we uh share the understanding that each of us holds so that we can you know talk about like what would because we talked about this in the past and I really liked Jared's question.

59:52He was asking, we're talking about like Kaizen Inc and we're talking about all this change, but are we actually improving? And that's why when I tried to think about this and I was thinking about like, okay, what would the improvement look like? And can we, I mean, we can measure it and we can check, have we delivered on this? And until like the last deploy that went out, I was fairly happy with the time that the duration that these deploys were taking. but based on the one which i have right in front of me the build going from one minute and a bit to almost three minutes i think that variability is something that i would like to understand first before optimizing the boot time is it the cpus then that's uh impacting it you think like the cpus and the horsepower behind the build test well let's open up uh namespace let's go to instances we can see the last build which you can see here like all the builds this is inside dagger is that right this is namespace all this is namespace by the way so we're using namespace for the runners and um i would like this is a third party service that's uh it is a third party service yes that you just found or someone told you about or exactly yes i am paying attention to various build services and you know depot.dev i love that namespace.so namespace.so yes our trial is almost over exactly yes now how much will it cost us by the way every minute three days left on your three days left with our trial yes i'm nervous i'm getting nervous here so so hang on per minute we're paying 0.0015 dollars which means that for 40 minutes like okay for an hour we're paying less than 10 cents for an hour of build time so you know pay as you go it's really not expensive so it's okay if i have to put my card because we're talking cents per month for our builds that makes sense what does a single build cost us then so when it's five minutes let's see i'll do the math now really quick hang on thank you hang on uh it's less than a cent a build which takes five minutes is less than as less than a cent is that right yeah less than a cent like 75 what is less than a cent zero cents no there was like another unit in the past i forget what it's called whatever like it's satoshi no that's a different the 70 75 percent of a cent okay okay so it's like reasonable it's reasonable yeah very very reasonable i would say if we get down faster it's even less exactly so what exactly does namespace do though i mean are they just is it just a machine that has proprietary code on it that we send something to to do a build process so namespace basically it runs custom github actions much quicker because they have better hardware better networking than github actions themselves so you can literally use namespace to replace github actions so they're just like they just use the actions api but you're running on their infra exactly smart or you can use like faster docker builds you know but they also have preview environments which i haven't tried and code sandboxes that's something sponsor i that's what I'm thinking because I have a shout out here and hang on let me just get the name straight to be clear they are not a sponsor but we're saying they should I think they should be I think uh Hugo I just know his first name and I'm trying to find um because our credit card is expiring right we need those six cents don't we we need those six cents that's Gerhard's credit card for now exactly you can use mine it's okay Hugo Santos no relation no relation yeah no relation no relation but I think if there is someone that you should talk at namespace I think it would be him like as I was like setting all this stuff up he was very responsive even on the weekend you know to emails and i think he's one of the founders by the way so i thought that was like a very nice touch and he really helped like go through all like the various questions which i had and the various like is does this look right so even like he even looked at the pull request to see how we implemented it and all in all like the promise is there we can see that it does work well when it works like two minutes, we get those two minutes, but sometimes it takes more.

1:04:02And then the question is, well, why does it take more? So that's something which I'm going to follow up on. Cool. Cool. Well, I am excited for the follow-up and for this progress. Indeed. Cool.

1:04:20Well, our friends over at SpeakEasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing, docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with Pydantic models for requests and response objects, an HTTPX client for async and synchronous method calls, and support for server-sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com slash Python.

1:05:05Again, speakeasy.com slash Python. And I'm also here with Todd Kaufman, CEO of Test Double. TestDouble.com. You may know Test Double from our good friend, Justin Serles. So Todd, on your homepage, I see an awesome quote from Eileen Yucateau. She says, quote, hot take, just have Test Double build all your stuff, end quote. We did not pay Eileen for that quote, to be clear, but we do very much appreciate her sharing it. Yeah, we had the great fortune to work with Eileen and Aaron Patterson on the upgrade of GitHub's Ruby Rails framework. And that's a relatively complex problem. It's a very large system.

1:05:45There's a lot of engineers actively working on it at the same time that we were performing that upgrade. So being able to collaborate with them, achieve the outcome of getting them upgraded to the latest and greatest Ruby on Rails that has all of the security patches and everything that you would expect of the more modern versions of the framework, while still not holding their business back from delivering features, we felt was a pretty significant accomplishment. And it's great to, you know, work with someone like Eileen and Aaron because we obviously learned a lot. We were able to collaborate effectively with them.

1:06:18But to hear that they were delighted by the outcome as well is very humbling for sure. Take me one layer deeper on this engagement. How many folks did you apply to this engagement? What was the objective? What did you do, et cetera? Yeah, I think we had between two and four people at any phase of the engagement. So we tend to run with relatively small teams. We do believe smaller teams tend to be more efficient and more productive. So wherever possible, we try to get by with as few people as we can. With this project, we were working directly with members from GitHub as well. So there were full-time staff on GitHub who were collaborating with us day in, day out on the project.

1:06:58This was a fairly clear set of expectations. We wanted to get to Rails, I believe 5.2 at the time, and Ruby like 2.5. Don't hold me to those numbers, but we had clear expectations at the outset. So from there, it was just a matter of figuring out the process that we were going to pursue to get these upgrades done without having a sizable impact on their team. A lot of the consultants on the project had some experience doing Rails upgrades, maybe not at that scale at that point. But it was really exciting because we were able to kind of develop a process that we think is very consistent in allowing Rails upgrades to be done without like providing a lot of risk to the client.

1:07:40So there's not a fear that, hey, we've missed something or, you know, this thing's going to fall over under scale. We do it very incrementally so that the team can, like I said, keep working on feature delivery without being impacted, but also so that we are very certain that we've covered all the bases and really got the system to a state where it's functionally equivalent to the last version, just on a newer version of Rails and Ruby. Very cool, Todd. I love it. Find out more about Test Double's software investment problem solvers at testdouble.com. that's testdouble.com t-e-s-t-d-o-u-b-l-e.com

1:08:25so i would like to switch gears to one of adam's questions and he was asking if neon is working for us as expected and the state of neon so is neon working for us as expected based on everything i've seen it is like i was looking at for example the metrics i was looking at how it behaves in the neon console this is us for the last 14 days so what we see here in the neon console we see our main database we can see that we have been using 0.04 percent of a cpu so really not cpu but in In terms of memory, we have eight allocated. We're using 1.3 gigabytes. 1.3 gigabytes used out of eight allocated.

1:09:10So we are over allocating both CPU and memory. So fairly little load, I would say, and things are just humming along. So no issues whatsoever. Do we need to push this harder somehow? Like, do we need to get the vector search in our database or something? Weren't you going to set us up an AI agent, Gerhard? Yes, I was. i didn't get to that but that would not use this database by the way that would be something different now pg vector man pg vector get it in there right i would but not in this production database so this is special right i mean this is well this is exactly what we want to see if anything we can because we have the minimum compute settings set to two cpus and eight gigs of memory and i know that um neon does an excellent job of auto scaling us when we need to we didn't need to get autoscaled because we are below the minimum threshold.

1:10:02So we could maybe even lower the threshold and it would still be fine. So we're not using this to its fullest extent, is my point. No. So we need some arbitrary workloads in order to push it. Well, to see where it breaks. We wouldn't need it to break. I think if anything, one thing that I would like us to do more is use Neon for development databases. And I have something there I haven't finished, but I would like to move on to that as well if everyone's fine. Adam, further thoughts or questions around Neon? This was your baby. I think the question I have is, you know, while the thresholds are low and we're below our over allocation, you know, what should we expect?

1:10:41And this is good news. This is good news that we're not. Yeah, I'm just saying like it's hard for us to use it and see if it's good or bad because we're not heavy database users. And I was just saying we just need some more arbitrary workloads to actually flex this thing. but I was mostly just being facetious. Gotcha. I'm in the dashboard too and I'm looking at a different section of that same monitoring section which is like rows. I believe rows being added which is kind of cool because over time you can kind of see your database updates essentially deleted, updated, inserted. So there's definitely obviously activity.

1:11:14We're aware of that. I think the other things that we should pay attention to in terms of is it working for us as expected I would say some of that is potentially on you, Jared, and you too, Gerhard, is that we've got the idea of branching. Jared, I know that you're familiar with it because you demonstrated some of this last time we talked, but being able to integrate some of those futuristic, let's just say, features into a database platform. This is managed, it's serverless. We don't have to manage it. We get a great dashboard. We get the opportunity for branches. Have you been using branches, Jared?

1:11:53Do you need to use branches? Does that workflow not matter to you? I think that's the DX and the performance is the two things I think I care about. So I have a custom branch, which I use to not develop against, but to sync from. I think, well, I guess it's not mine. It's that dev 2024. That's the one I use. Maybe Gerhard created that, but that's the one that I do use. Yeah. And so when I pull from that, So I'm not pulling from our main branch because there's just less load on our main branch to do that. And so I'm using that, but I synchronize it locally, manually, and then develop against my own Postgres because I have a local Postgres.

1:12:37It's not. The one thing about it is because it's a neon branch, I will have to go in here and synchronize it back up to the main and then pull from it. and I'm sure that's automatable but I just haven't done that. I've been waiting for Gerhard's all-in-one solution. Yes, that's coming. That's my next topic to share. What exactly is that? Well, that would mean tooling that's local to make all of this easy. Jared would need to go to the UI to click any buttons to do anything. He would just run a command locally and the thing would happen locally. He wouldn't need to even open this UI. Shouldn't that be a neon native thing though?

1:13:18It is. It does have a CLI. But the problem is you need to, first of all, install the CLI, configure the CLI, add quite a few flags, connect the CLI to your local Postgres, all that glue. That's the stuff which I've been working on. And I can talk about that a bit more. And so the idea would be to just automate some of that, not have to go through all the steps. Still do the CLI installation like any normal user would. Correct. but maybe a neon setup script that probably populates a file with credentials or something some command that yeah that you run locally that knows what the sequence of the steps is and what to do it's for example maybe you don't have the cli installed well install the cli you need to have some secrets well here's the one password cli and by the way the the secrets is here like in this vault so stuff like that like all that yeah speaking of one password did you knows their new SDKs.

1:14:14Did you pay attention to their new SDKs they deployed? TypeScript, Go, a couple others for native integrations. Obviously we're Elixir so it doesn't really matter to us but maybe in some of the Go pipelining I know you've probably done would it make sense to skip OP and go straight to Go with the SDK? Because OP is their CLI, right? It's same. It's not an SDK. The SDK lets you native integrate into the language. So it's possible to use something else. But at the end of the day, it's like the integration needs to work and the implementation, whether you use the SDK, whether you use the CLR, is just an implementation.

1:14:55Just doesn't matter, yeah. What we care about is like, is it reliable, our implementation? Do we have any issues with it? So far, no. Yeah. Are we using like service accounts? And that's something that we've been waiting because without service accounts, you would need to set up a connect server, which I didn't want to do. so that was a big deal for us whether you use the cli or the sdk we could but it wouldn't make that much of a difference now if the application itself while it runs it was doing certain things maybe that's interesting maybe we could get like change some of the the boot phase so that we wouldn't inject the secrets from outside the application the application itself could get them directly but i really want to get elixir releases going and once we have those you know things change a little bit but it's all just like maybe shuffling some code from here to here but ultimately it will still behave the same you know just like you would maybe bring it into the language so i haven't seen their latest sdks but i would like to check them out that's a good one for me to look into okay so the tooling that jared was mentioning to make things um simpler for him i've been thinking about it from from a couple of perspectives and i realized that to do this right it will be slightly harder and the reason why slightly harder is because i would like to challenge a status quo the status quo is you need a dagger for all of this maybe you don't right so i'm i'm trying a different approach and the mindset that i have for this is ken back september 2012 for each desired change make the change easy warning this may be hard then make the easy change so what i've done for this kaizen i made that change easy which was hard so that i could make the easy change how hard was it so that's what happened well let's have a look at it so we're looking now at pull request 521 and 521 introduces some new tooling but i promise it's just a cli and what's special about it is that everything runs locally there's no containers There's no Docker.

1:17:20There's no Dagger. Everything is local. And I can see Jared's eyebrows go up a bit because that's exactly what he wanted all this time. So what pull request 5.1 introduces is Just, which is a command runner. It's written in Rust, but it's just a CLI. And if you were, for example, Jared, or even Adam could try this. If you were to run Just in our repository, at the top level, you would see, just calls them recipes, what is possible. And the one which I think the audience will appreciate is just contribute. So remember how we had like this manual step, like install Postgres, you know, get Erlang, get Elixir, get this, get that.

1:18:06I mean, that's still valid, right? You can still use that manual approach. Or if you run just contribute, it will do all those things for you running local commands. It still uses Homebrew, it still uses asdf but everything that runs it runs it locally and the reason why this is cool is because i mean your local machine whatever you have running it remains king there's no containers again i keep mentioning this because that adds an extra layer and what that means stuff like for example importing a database in the local postgresql is simpler because that's what you already have running resolving the neon cli again it's just like a brewing stall it's there and you wire things together you don't have to deal with networking between containers you don't have to pass context inside of containers which can be tricky especially when it comes to sockets especially when it comes like to special files so i'm wondering how will this work out in practice and the thing which i didn't have time to do i didn't have time to implement just db-prod-import which would be the only command that you'd run to connect to neon pull down whatever needs to pull down maybe install the cli if it doesn't have it and then just in your local postgres import the latest copy same thing for just db fork which would be an equivalent of what we had before the differences that was all using dagger and containers and you know it was I mean, have you used it, Jared, apart from when we've done it?

1:19:43There you go. Adam, have you ever run Dagger? Never. In the three years that we've had it? Never. Not one time. There you go. How many times did you have to install things locally for you to be able to develop Changelog in the last three years? Well, that's where my personal angst relies. It lives right there in that question. How many times, what is the pain level that's high for me? So Adam might be more excited about this than I am. Pull request 5 to 1. I mean, even you, Jared, if you want to try it. I mean, if you do dry run, it has a dry run option, by the way. It won't apply anything, but it will show you all the commands that would run if you were to run them yourself, for example.

1:20:23And there may be quite a lot of stuff, right, when you look at it that way. But it's a good way to understand, like, if you were to do this locally, and if you were to configure all these things, what would it do without actually doing it? so i tried it on a brand new mac and i think that's the recording that i have um on that pull request i might need to get a brand new mac so i can try this look at that that that's a very very waiting for a good reason to upgrade you know there you go and honestly within five minutes depending on their internet connection everything should be set up everything is local the postgres everything what we don't yet have and i think this is where we're working towards is how do we first of all cleanse the data so that you know contributors can load a type of data locally but i think that's like a follow-up first of all we want jared to be able to do this with a single command refresh his local data and after i have this the bulk of the work done this step is really simple how simple maybe half an hour at most that's what i'm thinking so not much almost should be done before the day's over yeah should be done exactly should be done one thing i'm noticing is that you're switching back to brew install postgres i'm just curious about that change so i mentioned it in one of the comments when i committed basically when i was installing it via asdf the problem with was with icu4c i just couldn't compile postgres from asdf correctly and since then in homebrew we can now install postgres at 16.

1:22:02so you can specify which major version which was not possible i think two years ago when i did this initially so there is that now let's see let's see where this goes i'm excited about this if anyone tries it let us know how it goes for you uh if you want to contribute a changelog like how far does it get and by the i tested this on linux as well the easiest way there's like a something hidden there in the just it's called actions runner what it does is exactly what you think it does it runs a github actions runner locally for this you need docker by the way and it loads all the context of the repository inside of that runner so that's the beginning of what it would take to reuse this in the context of GitHub Actions.

1:22:51And what I'm wondering is, will it be faster than if we use Dagger? That's me challenging the status quo. The answer is either yes, it is, and maybe we should do that instead and it will shave off more time, or no, it's not. And then I get to finally upgrade Dagger because we're on a really old version. So you still work at Dagger, right? I do, yes, very much so, yes. Okay, I just want to know how much you want to challenge the status quo. No, no, no, that hasn't changed. I'm just kidding, I'm just kidding. Cool. Cool. So for our listener, if you want to try this, github.com slash the changelog slash changelog.com, clone the repo, brew install just, just contribute.

1:23:30That's it. Try those three steps if you're on Mac OS. If you're on Linux, it's not brew install just, it's apt get install or yum install or. The installations are there. Yeah. And just contribute. And what should we expect to see when we type in just contribute? is instructions or a set? No, no, it will actually run the commands. It's going to do it for you, man. If you do just dash n. Now, what if you have an existing repo like Adam does? Can he do it and it should pick up where he, yeah. Yeah. Give that a shot there, Adam. I'm so scared. What you could do is if you want maybe start a new user, it shouldn't mess anything up, to be honest.

1:24:09It just installs. Maybe it does things differently or does things twice. Yeah. I don't really know. But it should be safe. I like this. I mean, this is, I did run Just in our repository. You get contribute, depths, dev, install. These are all the actions or recipes. Correct. Install, Postgres down, Postgres up, tests. And each of those have a little hashtag next to it, which is a comment, essentially, of what the recipe does. So over time, we can expect to see more of these Just recipes if this pans out to be, you know, long term. these recipes will potentially get more and these will be a reliable way to do things within the repository and it's all local that's the big difference because before i mean even now right because we still kept dagger we still have everything that we that we had so far that would always run in containers which means it won't change anything locally and in some cases that's exactly what you want especially when you want to reduce the parity between test and production or staging and production but in this case it's local right so you want something to happen locally and local is not linux it means it's a mac so then you have that thing to deal with in which case brew helps and asdf helps and a couple of tools help but you still have to know what are the commands that you have to run in what order what needs to be present when and this basically captures all those commands it's a little bit like make which we had yeah right and we removed but this is a modern i would say version of that much simpler much more streamlined and a huge community around it.

1:25:45I was surprised to see how many people use Just. By the way, huge shout out to Casey, the author of Just. I really like what he did with the tool, like 20 ,000 stars on GitHub. A lot of releases, 114, fresh releases, 170 contributors. Yeah, it's a big ecosystem, I have to say. Without, one more question on this, without me having to read the docs, thank you, if you can help me on this. Can I do just dash n install so I can just see what it might, which is just the word just as many times, can I just see what it might do? Exactly. Okay. And dash n, it basically stands for dry run. Right. The reason why you have to do it before the recipe is because some recipes can have arguments.

1:26:29And if they don't, like if you do the dash n at the end, it won't work. So it has to be the command just, the flags, and then the recipe or recipes, because you can run multiple at once. Very cool. But yes. I assume that because like any good hacker that writes a CLI that's worth his weight in gold would always include a dash in, right? A dry run, yeah. Good job. What was his name? The maintainer? Casey. Let me see if I can pronounce his surname. He's Casey on GitHub, by the way. Rodarmore, the blue planet, apparently. Casey Rodarmore. You can correct us, Casey. Shout out to Casey. Yeah. C-A-S-E-Y.

1:27:09github.com slash c-a-s-e-y github.com slash I'm just kidding. I was going to say it one more time. Thanks Casey. Rod or more? Rod Armor. Rod Armor. Rod Armor. Yes, I like that. That's how we're pronouncing it. Casey Rod Armor. Correct us if that's correct or correct us if it's not correct or don't correct us. But go to github.com slash k-c-a-s-e-y Just do it. Just do it. Just do it. That's a good one. I like it. That's cool, man. Thank you for doing that. Not a problem. I enjoyed it. It was fun. Okay. HomeLab production. HomeLab to production. So next week on Wednesday, it's TalosCon and I'm calling it Justin's conference.

1:27:54It's the GarrisonCon. The GarrisonCon. Exactly. I'll finally meet Justin in person. I'm giving a talk. It's called HomeLab to production. It's, I think it's 5 PM. So one of the last ones we'll have a lot of fun i'm bringing my home lab to this conference so we will have fun i almost commented on that so it's not quite a home lab it's more of a mobile lab it is a mobile lab but i will have a router with me so it will be both the actual device and the router and yeah we'll have some fun now you're bringing two of them with you or just one the device the home lab plus the router so two devices okay well you want two of everything so yes well we are going into production so we're going to take all the workloads from the home lab and we're going to ship them into production during the talk and we're going to see how they work we're going to use talos kubernetes dagger is going to be there so yeah we'll have some fun this is a live demo then basically it's a live yes well it's recorded because you know i want to make sure that things will work but i will have the devices there with me you never know what wi-fi is like and that's the one thing which i don't want to risk yeah you can never even like 4G, 5G, even mobile networks are sometimes unreliable.

1:29:06But I'm looking forward to that. So that's like, and it will be a recorded talk as well. So yeah. Well, that's good because TalosCon is on-prem, free and co-located with SRE Day. However, it's also over with. By the time this ships, it'll be two days in the past. And so happy to hear, Gerhard, that there'll be a video because certainly our listener will want to see what you're up to and it's in the past tense. so there you go and guess what what i'm going to be recording myself as well okay what are you holding up there i'm holding a road pro do you know the road pros like the mini recording microphones yeah you can like clip them to your shirt something like that exactly so i have two of those boom and two cameras i'll take them with me the 361 so i'll be recording like the whole talk and then editing and publishing it so that's the plan cool so whatever the conference does great but I also want to do my own.

1:30:02Yeah. So that's the plan. Full control. Indeed. Awesome. Well, great conversation. Good progress this session. What do you call it? This Kaizen. This Kaizen, yes. What do we want to accomplish for the next one? Are we on the right trajectory? Like in terms of the things that we talked about, in terms of what we think is coming next? Did we miss anything? It'll be Christmas or just before Christmas. I think the just stuff with the database and branching with Jared being able to pull that down to be a small but big win. Okay. I think, you know, continue progress, obviously, on the pipe dream. Pipely.tech.

1:30:39Pipely.tech, I like it. Did you buy the domain? No, but it's available. I mean. Not available? It is available for 10 bucks. Pipely.tech. I don't know. I think we got to get pipe.ly. Otherwise. Yeah. Pipe.ly. We're just posers. But I like pipely.tech as well. So we might have to raise some money for this if we're going to have to buy pipe.ly. I might need 50 grand. The future's coming and we're going there. Kaizen. Kaizen. Bye friends.

1:31:12What do you think about our pipe dream? Should we turn it into a pipe reality? A pipely, if you will, let us know in Zulip. Yes, we are hanging out in Zulip now. It's so cool how we have it set up. Each podcast gets a channel and each episode becomes a topic. This is great because you no longer have to guess where to share your thoughts about a show. Even if you listen to an episode way later than everybody else, just find its topic and strike the conversation back up. There's a link in our show notes to join Changelog's Zulip. What are you waiting for? An engraved invitation? Hey, it's still September, which means we're still trading free changelog sticker packs for thoughtful five-star reviews and blog posts about our pods just send proof of your review to stickers at changelog.com along with your mailing address and we'll ship the goods directly to your mailbox anywhere in the world let's do this thanks once again to our partners at fly.io to our beat freaking residents the goat bmc and to our longtime sponsors at Sentry.

1:32:19Use code changelog when you sign up for the team plan and save yourself$100. That's almost four months free. Next week on the changelog, news on Monday, Ryan Dahl talking Dino 2 on Wednesday, and a fresh episode of changelog and friends on Friday. Have a great weekend. Leave us five-star reviews if you want some stickers, and let's talk again real soon.

1:32:53Game on!

From the publisher

Gerhard Lazu joins us for Kaizen 16! Our Pipe Dream™️ is becoming a reality, our custom feeds are shipping, our deploys are rolling out faster & our tooling is getting `just` right.

More from The Changelog: Software Development, Open Source

All 232 episodes
Forking Cal.com to closed source (Interview)3 Sep 2026 · 1 h 55 min
Postgres at PlanetScale (Interview)25 Aug 2026 · 1 h 42 min
Canary tokens and digital tripwires (Interview)21 Jul 2026 · 2 h 7 min
From open source hits to OpenAI (Interview)5 Jun 2026 · 1 h 46 min
Kaizen! Just do it (Friends)The Changelog: Software Development, Open Source · 1 h 33 min
Listen in VO