Kaizen! Mop-up job (Friends)

40:45And you do not expect a CDN to behave that way. And by the way, this is across both changelog.com and CDN changelog.com. So it includes both the static assets, everything. It's just a small window, but it just shows the problem. Now, as a percentage, that translates to 70.5%. So 70.5 % cash hits, and that is really not great. Okay, I know you've been expecting this. So let's see. What do you think is our current cash hit versus miss ratio? This is across all requests. So now that we switched across, we had 10 days to measure this. On the new system, what do you think is the cash hit versus miss ratio?

41:30Now you're giving us four choices. This is a multiple choice question. Only one. Only one is correct. 85 % is A, B is 89%, C is 95%, and D is 99%. Adam, what are you thinking? I'm locking in C. 95%. Okay. I'm going for the gusto. I'm going 99%. Whoa. Straight to D. I love to be wrong, but if I'm right, I'm going to be. Ah, I'm so wrong. 89.5%. So close, both of us were wrong. That's the answer. Yeah. How good would that have been, though? yeah well well you know some stuff is fresh it just is yeah i mean we we can improve on this we can i mean it's not now the important thing is it went from 70 to 90 okay that's right that was a big jump now and it's in our hands now we can actually affect exactly which is the last one before it was just like we could only complain now we can actually do stuff and even that after a few years of complaining you just become tired of complaining yeah yeah we grew weary we just we just build our own okay so can we do better we just can we do better than 89.5 percent but so i think everyone is thinking this but really what i think we should be thinking is do we need to do better do we need to do better than 89 okay so let's have a question feeds if you look at all the feeds we are at 99.5 cash hit ratio before they were at 96.8 so what are feeds for our listeners and watchers that know don't know who wants to answer that what are they yes what are these are xml files that represent the current state of our podcast syndication our product our are episodes that we're shipping and have shipped.

43:25So they're hit often by robots who are scraping feeds in order to update their podcast indexes and let people know which episodes are available. And they should be at 99.5 % because they only change when we publish a new episode, which is at this point in our lives three times a week, on a Monday, on a Wednesday, and on a Friday. And every other request, every other day and time is the same exact content. That's it. So I would say that this is possibly the most important, or yeah, I would say the most important thing to serve. Because if we don't serve fees correctly, how do you know what content changelog has?

44:04How do you know when content updates? And this is like worldwide. So I think this is pretty good. And improving on 99.5%, I don't think we should do it. The homepage before, the hit ratio was 18.8%. Oh my gosh. I know. This was my biggest issue for as long as I can remember. Today, it's 98.5%. And to our listener who's probably out there thinking, you guys were certainly doing it wrong. We spent years trying to do it differently. Go back and listen to all the Kaizens of us trying to change the way that we actually configure and respond and our headers. and our, I mean, we tried and we ended up with 18.8%.

44:52That's it. Cache hits on our homepage. And so here we are. Oh, so bad. So bad. Now, MP3s, I would say the second most important thing is 86%. Now it's 87.5%. Maybe this can be better. Maybe this can be improved. We just need basically more memory or store them on disk, do a bunch of things. Because by the way, caching is in memory. And I still need to understand why memory, once it gets filled, it doesn't remain filled. So I've seen this weird behavior where after like a few hours, memory starts dropping. But why? Because there's no pressure on memory. Why are objects getting evicted? My assumption is that we're storing some large objects.

45:35And if there are smaller objects that need to be stored in memory, the larger objects get evicted, which means that the cache drops. but I would need to understand that a little bit better. Still, MP3s aren't worse than they were before. News, I know this is something that's very important to Jared. It was 52.6 % cached before. Now it's 83%. So improvements across the board. Now we could improve it. And I think we, especially news and MP3s, I would like to look into that. But I think news is like top of my list. But is there anything else that you think that we should pay attention to or in terms of the cash hit ratio?

46:19Any other resources? No, I mean, MP3s are static assets. You could go look at other static assets, images, etc. But I just don't think we want to squeeze this radish too hard. I agree that news and probably taking a low-hanging fruit pass on the MP3 endpoints and seeing what we could do there would probably be bear would bear some good fruit but even those i wouldn't like spend hours and hours trying to make them much better what i'm thinking i would just basically double up the memory and see how the changes things which is just the config setting it'll take me maybe a minute that that will be my first action item okay okay yeah i imagine news could be very similar to feeds because once news is published it's similar to the feed it's not really it's changing once a week and once it's published it likely never changes again it never has changed right now that we've moved all of our comments and everything exists elsewhere i mean they're in zulip they're on youtube so there's no comment feed there there's no reason for anything to really be new right except on publish yeah okay so i'd say the news is the one of the feed pushing that to the boundary because it doesn't change much i'd love to explore that when you do the mp3 exploration of large objects getting pushed out i'd love to just sit on your shoulder i suppose or as a fly on the wall kind of thing just explore that with you because i'm super curious about what makes that cache get purged out of the memory myself yeah well pairing up is something that i'm getting better and better every day recorded and published pairing sessions so jared has the Experience, not Adam.

48:03I'm sorry. Yeah, that sounds great. Now, it gets better. Oh my God. Okay. It gets better. What does? All of it. Do you recognize this seed? Okay, so that looks like - This is Johnny Mnemonic, right? No. Say again? Johnny Mnemonic, is this - Nope. No, that's - Is that Hugh Jackman? Yes. This is not - Yeah, this is not Wolverine. Swordfish. This is Swordfish. Oh no, I'm not sure. I'm very nervous right now, Gerhard. That's okay. It's fine. It's recorded. Swordfish broke the show previously. Okay. I'm ready for it. That doesn't mean that I'm ready for it. You're going to play this live for me right now on camera.

48:54Okay. I will, yes. So this clip is going to blow you away. I'm going to watch it. Okay. I'm hoping this is Sora.

49:06Okay, Halle Berry, John Travolta. Jackman has a gun to his head. He's typing. He has to hack something in a certain amount of time. Was it 30 seconds? 45 seconds. 45 seconds. 45. Look at his fingers. Types very, very furiously. He's typing furiously. Oh, access to the knife. He gets very disappointed when that happens. He gets very disappointed. Yes. what axis gets he died all right okay so um that was that was a moment of fun it was it was it was it's things get better okay okay so that's a moment of disappointment exactly even if you're under pressure you and you have to deliver things will be better okay so pipe dream gets better okay now we looked at the cash it ratio what i would like to look at next is the response time in seconds okay these are all the feed requests before and after so the p50 for all the feed requests used to be two milliseconds and you would think wow that's pretty good well the new system is like half a millisecond so it's a four-time improvement the p75 is 13 times better so for 75 of users 75 of the users the feed responses get served 13 times as quickly as they were before the p199599 like it gets progressively better which means that the requests are served much quicker at least four times as quick as they were before and you might be thinking hey that's bots what about humans so what about the home page for 50 percent of the users the home page is 860 times quicker that's nearly three orders of magnitude quicker that's a crazy amount quicker now obviously it is the fact that it was not cached like only 18 of the requests were cached right but the page is instant like now it's instant it's 0.000 like three zeros three seconds that's like a third of a millisecond that's nice so you're welcome humans you're welcome humans now what does that look like i think 863 times is really difficult to imagine so i'm going to i'm going to play something for you to see what it means so what we have here is one second at the top that's how long it takes no hang on i'm not playing it i should be playing it there you go now i'm playing it okay while 833 seconds at the bottom is still loading and it will continue loading for so long that we're not going to wait 15 minutes for this thing to load okay we're not going to wait that so that's the difference between how fast the home page loads now versus how it used to load before this is for the majority of the users so the cache hit ratio the connection there was that everything was slow and there's nothing we could do about it and i think slow is relative right because when you think when you're talking about milliseconds um i think there's about 50 or maybe 100 milliseconds when things are nearly instant but in our case the home page was taking 150 milliseconds to get served and the more like the tail latencies were really crazy like the tail latency was over a second for the home page to serve that was a long time by the way this thing is still going and it's not even like 10 % there.

52:58What's the rationale behind this video? Explain to me how this is supposed to explain things. So the top one shows you how quickly it takes for one second to finish, right? So the response, the one second response, it just visualizes it how quickly that gets served. The bottom one shows you how long the previous CD and how long in comparison 863 seconds is. So we have now, like things are loading in a second or the equivalent of a second. And before things were taking 863 seconds to load. They were? The same request. Longer. Relative, not absolute. Relative. Okay. Exactly. So the one second represents our current loading speed, which is like a millisecond.

53:52but we can't visualize that because it's too fast. And 863 seconds, that's how much slower it used to be. So it's a relative example. Relative to one second versus relative to milliseconds. Right. That way we can actually visualize it. And now it's at two point. Oh, he reset it. Okay. I was going to say, it's going backwards. It's like loading again because it's finished. So it's like 15 minutes versus one second and then reduce that down to milliseconds and we're basically that much faster. Exactly. 15 yeah waiting 15 minutes versus waiting a second that's exactly the speed difference between what we used to have and what we have now right it's just really fast and that was that was really really fast that was a cdn thing that was not that was traversing dns into cdn getting a cache hit or miss serving you know rehydrating getting new from cache that's where all the time was spent So in this case, these were the responses from the Varnish perspective.

54:52So if I go back here and we're looking at this table, so this is the homepage, the response time. How long does it take to serve the homepage from Varnish's perspective? Whether it's in the cache, whether it needs to go to the application and request it and then eventually finish serving the actual response. so before p50 259 milliseconds that's how long it used to take yeah and now it takes a third of a millisecond what changed specifically with varnish then well caching most of the like if we go back to the table if you go back here now 98.5 of the requests are served from cache We don't need to go back.

55:39I mean, the homepage is almost always in cache. We very rarely have to go back to the application to fulfill a request. Before, only 18 % or 19 % of the requests could be served from cache. So Varnish had to go to the application. The application had to serve the response so that Varnish could serve the response back to the user, to the end user. You keep saying Varnish, don't you mean vinyl? Yes, I do. I do. Not that you just have to tell you, but there's a. Vinyl, yeah, vinyl is coming up in. There's a curveball. Yeah, in January. So it's not, it's not here yet. For those not in the know, they're renaming the Varnish software.

56:18They are, yeah. Because of legal disputes to Vinyl Cash. So Varnish Cash, the open source project, will be renamed to Vinyl Cash. That's correct. Whereas Varnish Software, the company, will continue as Varnish Software, the company. I'm going to jump ahead. I mean, this is like, yeah, Jared is coming from the future now. I am. So PHK, Poole Henningkamp, he posted on September 15th. This is on varnishcash.org. It was just about a month ago. He wrote the 20 years old, 20 years old, and it is time to get serious, sir. That is the title of the blog post. And he talks about the open source, yeah, the open source varnish cash rename.

56:58Some legal disputes, indeed. You can go and read it out. But basically Varnish and 8.0, the open source Varnish, 8.0 will be the last one that will be called Varnish Cache. So from March next year, and I think this is very interesting, it will be Vinyl. Name of Varnish, the open source Varnish will get renamed to Vinyl. Okay, now I know who you have to have as a guest on our interview in Austin. PHK launching the Vinyl Cache, live on stage. I think he lives over on your side of the ocean. Or Berlin, yeah, because he is. He probably wouldn't come to Austin, but we can try. So that's, yeah. All right.

57:42So speaking of speed, this one's for Adam. Love speed. You know that I joined Loophole Labs. That's it. So what we do, it requires a really fast, low latency network. And by that, I mean at least 100 gigabits. Okay, it needs to have sub one millisecond latency. So about two months ago, I started building a new 100 gigabit home lab. And I know that Adam has been asking me for a really long time to do video on my home lab. So Jared already watched it a few weeks ago. This is live. And these are ways to go to yt.makeitwork.tv. That will take you to YouTube and there you can go and watch it. And I take you through the entire journey, why I had to build it, a couple of interesting things.

58:26Yeah, it's, how did you, I mean, what did you think about it, Jared? the portions that you managed to watch. Yeah, like I said earlier, I thought it was really good. It's cool. What did it miss? What did it miss? How would I improve it? Maybe that's a better question. What would have made the video better? What would have made the video better? I don't know in terms of like content that was missing. I feel like the way that you do your summarized. intro is compelling, but sometimes it jump cuts so fast. or it's sometimes hard for me to track exactly. So that's like a hard thing to give feedback on.

59:08I'd have to give you specifics for you to actually know what you're, but you do, even though I'm like compelled to continue watching and I do enjoy it, there are times where I'm confused. And so I think as you continue to refine that, because I know it's a style that you're doing and you've gotten better at it since, because I've seen some of your videos from a year ago-ish. And it's definitely getting there where I'm like, I look forward to it. But I also sometimes am not sure what's going on. I'm not sure if that's on purpose. Maybe it is to keep me intrigued as there are all kinds of techniques to keep people watching on social media.

59:39But I think you can continue to refine the narrative that kicks off from the beginning in, in making it more cohesive or just, you know, laying more breadcrumbs perhaps for the person who's not initiated. Cause you're so deep into what happened and you know, the whole story from front to back and you're telling it and that's great but i i think that's one thing it could be improved um and first i don't know about stuff that's missing but because i wasn't there for what miss what you know what got dropped that's great that that's great so yeah okay that's excellent feedback all right well adam when you get a chance to watch it if you get a chance to watch it i would love uh to get some feedback from you too into what in terms of what would make it better for you for sure as you watch it anything that could be improved because the next one is coming up so how fast is the pipe dream that's what we're going to answer now how fast is the pipe dream on this 100 gigabit home lab because you remember we talked before uh we're looking at um how much like when we benchmark these things fly.io itself is limiting us to how much bandwidth we can push right we wouldn't want to be pushing tens of gigabytes or hundreds of gigabytes that would be crazy because that's cost someone money so running benchmarks like that is this is not great but also um the wan there's like a limit there but in this case the limit is 100 gigabit we know that so this is a new home lab this is what it looks like it's running ubuntu 2404 um i think the most interesting thing is the cpus about it so it has the threadripper 9970X.

1:01:23And the reason for the Threadripper is because I need a lot of PCIe lanes. So I couldn't use like a regular consumer grade CPU. I needed 16 lanes for the motherboard, for the GPU, the RTX 4080, which is there at the bottom. And I need another 16 lanes for the network card. Even though it's a PCIe 3.0, if you give it, for example, eight lanes of, in this case, PCIe 5, it can only use eight lanes of pci3 so you would like basically half its its its speed and eight lanes of pci3.0 means 64 gigabits so really i need a full 16 lanes doesn't matter i mean needs to be at least pci3 to achieve its full speed and the card is like the thing that you see there's some green led lights it's right below the fan for the cpu all right so that's one this is the server we're going to be running pipetreem on this host and this is the other part of the home lab this is my older machine it's about three years now it's a ryzen 7 5800x it has 16 threads not 16 core 16 threads um a small gpu g4 gt730 is a fanless one it's the one at the bottom and again the reason why i had to do this is because i needed the 16 lanes which are right under again that cpu cooler for the network card if i was for example to put an nvme in a specific slot or if i was going to fill any other pcie slots the first slot because it shares bandwidth the first pc slot would be limited to 8x which again would create that 16 sorry that 64 gigabits limit so i needed to give this full 16 lanes and that's where this limitation came from now this is the star of the show this is what that card look like those cards look like um you can see i mean there's like a DAC cable obviously you need two now they have two modules which is interesting because the card itself in terms of the PCIe bandwidth will max out at 128 gigabits so it can't go beyond that from a PCIe slot perspective even though each module could do 100 gigabits really in combination you can only push 128 theoretical maximum and these are older cards it's a it's an nvidia melanox connect x5 so they've been around for many years now uh i think 2020 at this point so they're like about five year old cards what's the price range on these are they're expensive uh they can be yes these specific ones i got them off ebay and i paid for both of them 500 pounds which i think it's about eight hundred dollars i think seven eight hundred dollars roughly um now in the us i think you can get them for even cheaper because you get more hardware right data center hardware that just gets you know sold for um a good price so refurbished stuff or in this case it's just used it's not refurbished and a DAC cable the cable can be a bit expensive you can pay anywhere between close to a hundred dollars i think this one was about 60 60 pounds and then you need two um and the reason why you need two is here because i'm configuring the two modules in a bond in a network bond it's an lacp bond and in this case what it means is that i'm basically having two cables and i'm creating a single virtual connection that uses both um both modules so again the theoretical maximum is 128 gigabits but in reality it's more like 112 i was not able to push it beyond that so http stat what i'm showing here is that if i do an http stat from the client h22 by the way h22 is the client that's the year homelab 2022 and w25 the workstation 25 but it's also homelab but it's also workstation so it's a it's a combined thing because of the the many cores that it has and it has a multi multi-role that host all right so from the client from h22 i'm basically going to that private ip uh 1025 10141 and on port 9000 the pipe dream is running and you can see here the response is going to the app it's just basically a pipe dream running in a container locally as it would run on fly all right so let's see what this baby can do i'm using oha i'm running 64 clients and i'm sending a million requests to the home page let's see what it will do

1:06:05that was it that was one million requests sent to the home page oh my gosh so it took less than seconds to complete we pushed 225 000 requests per second in terms of data we transferred 12.5 gigabytes in five seconds wow we reached 2.8 gigabytes per second and that's over just over 22 gigabits per second so if you were to guess what would you say the bottleneck is in this case you just had to guess it's not the network we know it's not the network it has to be the cpu yeah it's actually yeah so the container i mean it doesn't have it has it doesn't have any overhead it uses the it uses it binds to the local network so there's like no uh no natting no bridging um nothing happening from a networking perspective it binds to a port on the local networks when you go to 9000 it's the port local 9000 on the host now I'm wondering what happens if you fetch the master feed because the master feed is really big it's 13 megabytes in size so it's about a hundred times the size of the home page so we're going to do something very similar the difference is we're only going to run a hundred thousand requests not a million requests and this is what that looks like okay it's taking a little bit longer what do we see what do we see here so here we see the cpu and the network for the client h22 and what stands out is that many of the cores are 90 80 close to 90 usage we can see the network throughput it's 4.59 gigabytes per second which in this case is 36 gigabits so we're getting close to 40 gigabits per second but in this case we can definitely see that it is the cpu that seems to be the bottleneck but the cpu on the client so this is where we're running the benchmark from so i'm wondering what does the cpu look like on the host on the big one the thread ripper the thread ripper it looks pretty quiet from this pretty quiet pretty chill that's it yeah i think the peak that there's one core which is 11 but otherwise everything is like less than even five percent so most of the cores are chilling so this confirms that we have the bandwidth i mean we went from 20 to about 40 gigabytes, gigabits per second, from 20 gigabits to 40 gigabits per second.

1:08:56And we can see the CPU is fine. So PyDream could serve more from the instance perspective, but the client where Oha runs, the one that benchmarks, seems to be approaching a limit. Even so, we are able to send 100 ,000 requests. And I think it takes, I know, about 40 seconds, something like that, roughly. We can see again the host, the client, which is running really really hot 36 constant so that's nice and constant and we'll see it now at the end finish 100 000 requests it transferred about 40 something gigabytes this is where like this in terms of traffic it would cost a few dollars just this benchmark we transferred 200 gigabytes of data for this one and the peak the request per second was 2200 that's pretty good that's pretty good so i'm wondering what would happen there's like one more thing which i would like to do what would happen if we move the client from the host with the slower cpus to the host with the faster cpus right so we do like a swap around we run pipe dream on the slower host but we run the client that benchmarks things on the faster host so what would that look like all right so same hundred thousand requests but now we've reversed where we run these things and this is what it looks like it's looking faster this is the client this is where varnish runs sorry this is where the pipe dream runs the cpus are 100 basically and this is where a benchmark runs we see some um yeah this is really so let me just go a little bit back there you go so we see 80 gigabits per second now.

1:10:4280 gigabits per second, yes. So we're able to nearly achieve like the performance, like the saturate the network. And we can see the CPUs. We still have plenty of CPU room on the New York station, the Threadripper, which has 64 threads. So plenty of CPUs there. Which is now running in OHA. It's not running Pypream. Not Pypream. But we haven't maxed the client out. Now we're actually maxing the server out. That's it. So in this case, if this host had faster CPUs, it could go faster. But now we're just basically bottlenecked on the CPU. Well, you're going to have to upgrade your host, Gerhard. I think I will.

1:11:21We need answers. We need answers. I think we will. But this just goes to show that the setup scales really, really nicely. And this is what we've been all working towards. Fly.io and Vinyl and Change.Log.io. so sorry changelog.com so it's a good combo um in this case honestly fly is sometimes like uh throttling us on on the bandwidth that makes sense i don't think they were expecting a cdn to run on fly honestly i mean some of the peaks we can be pushing up to 10 gigabits i've seen like in in the metrics so far there's these are like only like the peaks i don't know what the limit is on fly but i think talking to them about this would be a good idea i've been mentioning this but i think that's following up on this because i mean what happens if there's i mean maybe throttling is what we want but it will affect other users so are they okay with us running a cdn what can we expect from the setup so yeah so vinyl we already know we've been here the vinyl is the rename varnish will become the open source varnish cache will become vinyl from march next year so by the time we meet next pipely and pipe dream will be vinyl all right we're closely approaching the wrap-up what's next let's talk about what's next the first thing on my list is I want my, my, my, my bam.

1:12:55What does bam mean? Oh, your big ass monitor. That's it. The big ass monitor. So now when I switch my big ass monitor behind me, what I see is all of Pipe Dream. All of it. I see all the traffic going through changelog. It's beautiful. Look at it. I see all the areas which are like the hot. It's just so nice. Seriously. This is the best painting I could hang. in my study and i have it on all the time so i just see it it refreshes periodically this is the fly.io dashboard on the left i have the edge how does the edge behave this is the fly proxy for our cdn and on the right depending on how you're looking at it on the right uh i see the fly app which is in this case a pipe dream itself in terms of memory usage cpu usage all those things and uh it's a thing of beauty now i understand when there's a problem i see the the memory when it drops like all the things are just there and it works well so the bam is done that was a quick one and thank you fly for a great dashboard that was really good um but the one thing that keeps coming up is out of memory crashes they happen rarely but they still happened and even though we have the limit setup i need to understand what exactly triggers those crashes how to prevent them from happening so sometimes um and i say sometimes i think the last time it happened maybe a week ago two weeks ago it's it hasn't been this week so an instance when it gets overloaded it runs out of memory it crashes all it means is that the requests they get routed to a different instance and when the instance restarts it starts like with an empty cache so takes a while for the, for the, for the cache to fill in memory.

1:14:41So I'd like to dig into that. Remember the logs, our events, our metrics, Jared, that we've been talking about, where currently they're being stored in S3. And then you have the job, a cron job, which processes them out. I'd really like to sort out that pipeline because part of pipetream, part of rolling this out, I just realized how everything is put together basically. and i think we can improve on that so what i'm yeah what i'm thinking is if we were able to ship all the logs um in a in a in a column like format so like a white format in something like click house and i'm thinking click house specifically this would just make everything so much easier like the whole metrics analytics pipeline that we have not to mention that in a way that would be two of everything and honeycomb could be one of them but click house if it stores all these requests it could be our other um event store right and we can visualize all those events so that would be good now do we know anyone at click house at click house cloud specifically do we have any click house cloud friends danny we talked to danny a couple years ago remember that at open source summit in Vancouver.

1:16:01I don't remember that, but that's why there's two of us. I'm pretty sure we know some folks, if not through acquisition directly. So we'll hunt our contacts and come back to you. So ClickHouse would be really interesting or anything that can store lots of events because every single request in our case would be stored there. We would batch them and do all of that. but we would write quite a few events. I mean, it's basically every single request would end up in that data store and then we'd need to be able to aggregate them and read them back really, really quickly, which I think would do away with any of the analytics eventually, but we're currently doing Postgres with the background jobs.

1:16:45All of that can just go. Come from that. Exactly. Yeah, the challenge there is that, what stack on ClickHouse in particular? Because that's tying us to yet another behemoth that might make us hold it wrong, potentially, and we'll be forced to build something else. ClickHouse spelled H-A-U-S instead of H-O-U-S, you know, the German version of it. What exactly do you like about that flow? So what makes that be the first-class citizen for you for data transporting? So I've used it for a couple of years. We've been using it at Dagger and continue doing so. And it works, it works really well at a large scale.

1:17:31So it processes not billions, like trillions of events. It scales really nicely and it's really fast. And the ClickHouse Cloud team specifically, they've been very supportive and they seem to be innovating and doing things in a very thorough way. So it's something that has always been dependable. Now, we currently use Honeycomb for like the whole UI thing, but we also store a subset of the requests in S3 and then we process them and I think we store them in Postgres. So we duplicate these things in a couple of places. If we had a single place where we store them and this could be like ClickHouse, that would be like the primary store, we could read any metrics whenever we need we can create materialized views i was like it's just so flexible in terms of how we can slice it and dice it it would be our alternative to honeycomb and we still love them and i definitely see us continuing using them but it wouldn't be the only one um and we would have the same view in terms for alerting or monitoring or anything like that we'll need to do something separate so it just centralizes every single request coming from every single instance in this case yeah and obviously we still store them in s3 clickhouse can read from s3 which is really nice it has support like a special parquet format which means that you can have like like data store like in long long-term storage in an s3 like um system and then if whatever was to happen with clickhouse it's down or there's a problem with it it hardly ever happens.

1:19:13I mean, I think I've seen it only happen once in like almost four years. So it's been very reliable in that way. But it means that we can, we can revamp how we do analytics. I'm not sure how you think about that, Jared, because I mean, you've been mostly using that, like the analytics that you get in the app. How does it, I mean, do you need to upgrade it? Do you need to work with it like is it always like do you just forget about it mostly you set it up and you forgot about it mostly forgot about it i think the biggest drawback is how infrequently it updates at this point and so we don't i mean maybe that's a feature because we don't like check our analytics um on the daily on the secondly obsessively like maybe we could that being said if we can get that information faster and learn something along the way i think it's worth it and i'm certain that this this would change the way that we do things enough that we could get that information much faster than via a cron job so i'm interested in it would it bring huge value to us to be able to see the downloads faster probably not because we've habitually not done that you know we just check in on every once in a while but i'm up for learning and trying and improving.

1:20:35So I definitely think it's worth the R &D budget. Okay. I mean, again, it comes as a suggestion. I think we've been doing this for such a long time in terms of discussing these things in public and I think this keeps coming up. We can defer it. It doesn't need to happen. It's just something that I had to touch in the context of the pipe dream as I was looking at where the metrics are going, the feed requests, how is that like put together what do we write in s3 all all those things like the different buckets like all that stuff i had to go through basically part part of part of this i would love to remove just to chop that part off it's just there as a thing that we do you know but yeah doesn't have to be that way okay that's just how we did it so do you have a thing that you'd like to improve between now and the next kaizen either of you because i can go i can keep going through this list, but I'm wondering if there's something that you're thinking about or something that's bugging you that you would like to see improved.

1:21:36I've kind of just become content of late with the way things are. Adam? Okay. Hmm. I think the only thing I think about really is when I think about this system, it's kind of sad in a way, is that if eventually, and it's not the case currently, But if eventually the case is that our true traffic comes from one of the spokes versus the hub, like how important does the hub remain? And the one thing we're not tracking really is the MP4 file that we upload to YouTube in terms of the system, right? So we don't have a store. So we have the MP3 for the plus plus version and the public version side by side in perpetuity for all of time back to episode one of all podcasts.

1:22:29What we don't have is that corresponding video file because it's not part of this pipeline. It's not part of the serving pipeline. And that's what I think about. And I think about the analytics and the effort there. you know how does how does the the change to watching viewing listening patterns change the show over time change the system you know that's what i think about so i'm not really sure i've certainly considered us a modification where we upload the mp4 to our system yeah and let it redistribute to the various places that we wanted to live that's a much heavier lift and i think there's benefit at the moment at least i agree yeah we still have the mp4 so it's not like we don't own that content we just don't own it you know on r2 alongside our mp3s and it's kind of a sad spend of money to store a file you never really you send a youtube one time right you know so i can also start to get like diverse with it and say well we're also going to be on peer tube and yeah we're going to upload to rumble or i don't know what all these places are now because people kind of scatter from YouTube and then they gather again and they scatter.

1:23:42But so far, the point of our video is to be on YouTube at the moment. And so we haven't thought beyond that. But if YouTube changes in some sort of dramatic way where, I mean, we've seen changes over the time where it's like, let's be in more places, similar to how we're not just on Twitter slash X anymore. We're on more social networks. There may be a day where that happens with YouTube and we'd be happy to have the pipeline and set up to where we can just, you know, add another connector to it and say, we're also on this video watching platform, or we can view our stuff directly, you know, and watch it directly on our website and we'll let Cloudflare bear that burden.

1:24:21Although if it's passing through PipeDream, we're going to have some serious bandwidth going through Fly.io. So those are things I've thought of, but I've never been even, at this point, I'm not even close to pulling that trigger. Gerhard, you're on YouTube now, but you also have MakeItWork.tv. So you're actually tackling this to a certain extent. How do you do it? Jellyfin, right? That's one. Yes, Jellyfin is if you have a client, that's right, you connect to the server and you can download things. You can store them like offline on your device. So it's like a media library, proper media library in a media server.

1:24:54I find that works really well. I mean, I always prefer watching it that way. But I also store it on a CD, in this case, Bunny. so bunny has uh stream i think it's called and i think cloudler has stream as well uh where you just basically upload um um media content in this case uh mp4 files um so the cdn part works well i would love to replace that to be honest because i'm not entirely happy with how the system works the chapters are a bit clunky there's quite a few things that are clunky the the trick play isn't great again lots and lots of things just the way you upload things is just just just too much work so i'd love to um automate that but youtube really is like the main distribution mechanism not just because of how easy it makes it to just upload the file and then gets redistributed everywhere but also it's almost like you have something to sell do you go on ebay or do you go to flea market or elsewhere do you build like your own shop to sell that thing and ebay a lot of the time is easy because that's where the a lot of the buyers are that's where people are looking and searching or craig craig's list whatever they could gumtree we had it in the uk i think it's still a thing so youtube is a place where lots and lots of people already are so in terms of distribution it just makes it so easy um now i'd like to give the option of hey if you if you don't want youtube that's okay you can also download it from the cd and that i pay for that i set up and in this case i haven't built but that's coming so or or jellyfin now i don't know how many users would set up a jellyfin to be honest um for changelog but i like the idea of basically having that one-to-one relationship between the creator and the watcher the listener the viewer there's like nothing in between so youtube can't push its ads youtube can't ban certain content in certain places if if it happens however it happens um again it's very difficult to know that because you need to be in those places know how that works and also like the idea of people being able to download the content and again the podcast players make podcast players makes it easy for youtube you have to pay that premium you have to go youtube premium which i do and i have for many years.

1:27:17Um, and I think it works great, but again, my experience, my YouTube experience is very different to most people because I don't think many people pay for YouTube. It's just like an extra expense. Um, so that's my take. Yeah. It'd be cool if there was a vibrant community of people that consumed video via open standards, like they do with podcasts. I mean, the, the coolest thing about podcasting is that phrase, you know, get this wherever you get your podcasts. And it's like, that's because it's an open standard that you can just directly subscribe to a feed and people can build apps for those.

1:27:56And that's amazing. That doesn't exist for video. Will it someday? Maybe there are nerds out there and we've, we get the emails about open standards for video podcasts. And actually Apple launched the iTunes podcast section with video podcasts as like a first class citizen. It's just that nobody, the bandwidth was so expensive back then. This is like either pre-YouTube or like right around the time that YouTube started. And like people were just weren't watching. They didn't have the, we just didn't have the technology to actually make that a thing that you just watched. You didn't have the phone.

1:28:30You had to like move the files around. They were large. I think MacBreak Weekly was like one of the only ones or whatever saw, they were like shipping 4K video podcasts like in 2007 or something, and it was crazy. Wow, that's hardcore. Because they come from the TV side, you know, where they're used to putting out video and where most of us are just coming from the audio side. Anyways, those things existed. Apple obviously just kind of like, it still actually is part of their RSS spec, but no one uses it. Even Apple Podcasts, I'm not sure if it even uses it. Spotify has their own deal for video.

1:29:02It's not using the open way. It's using their own proprietary way. There's weirdness there where if your video file duration or details differ from your audio file, you may end up serving one or the other, even to audio listeners that you don't want to. And so it's just kind of murky right now. And I think it would take some sort of a black swan event and maybe a sea change in opinion and some sort of new tech that makes it feasible, at which point I'd be all about it. I just think right now it's like a lot of effort. and there's a lot of effort to like re-encode your video into all these different formats depending on blah blah blah and then serve that you know a lot of cash misses if you got six versions of your video depending on the client and so it's like money the storage too yeah it's like money time and effort for right now like a very minuscule advantage let youtube pay that price and their tech and their tech stack and their developers and their bandwidth and their servers etc yeah that's that's my only concern is that really is is like where do we begin where do we reach diminishing returns in innovation you know with the cdn beyond mp3 and smaller file it doesn't seem to naturally scale to the video because the incumbents have that solved in ways we just don't see that we need to solve those problems the only problem i guess that i see is the fact that we don't have this video file artifact alongside the mp3 artifact that is the same thing but a different flavor of it yeah it's elsewhere in our archiving stack and i would say largely inaccessible certainly not via an api you know potentially but it's not right i think building a standard or contributing towards the standards takes a really long time yeah lots of effort and that's why no one wants to do it and they're waiting for someone else to do it because they know how much of how much investment that takes no one's in the mood for that type of investment right um i think it's going to be very interesting what happens with ai because there's a lot of money right now in ai and i think it will change not before long so where will that money go next we'll see um i'm definitely curious for for the next big thing which is coming but um Um, one thing that may work well is if changelog had an app, had a native app, whether it's an Android or iOS app, and then you control how you display the video and the MP3.

1:31:40And if you had something like that, then you'd be in full control how you could, how you would expose your MP4 files and how you would integrate them in the player. And that's like a more holistic experience where, you know, you can do like transcripts really well. You can do comments really well, maybe integrated with some sort of Zulip or something like that, where it feels like a system and it's more like a community of people that are interested in this type of things rather than, you know, just some content that gets distributed on different platforms. Again, the problem in that case is that most people are already on those platforms.

1:32:17So it's easy for them to consume things, but there will come a point where they just get they i mean they just want something different the flow plane i mean that's that's a new thing that's been being run for a few years so that's one example there's another one i forget what it's called i know that you can you can pay for man i wish i knew i remember the name i was doing a research maybe like six months ago more like nine months ago actually was more than six months ago and i was looking at youtube alternatives and there's like this other platform for media, which stores and distributes higher and like 4K videos, 8K videos.

1:32:55But for that, you end up paying. So it's not free. And then you get creators that publish only on that platform. I forget its name. I can look it up. Maybe we can add in the show notes because I have it somewhere. But that is another interesting thing. Vimeo. I remember when Vimeo was a thing, but I still, I know it's around, but I don't think many still use it in terms of like people going and browsing. I don't think it's even a thing anymore. Yeah, they pivoted quite a bit. It's a very successful business to this day, but it's like serving enterprise and more professionals who are using videos for various purposes, not as a general consumer product at all.

1:33:35And video is hard because the transcoding part is really hard. and um yeah i mean you always have like the trade-off do you pay storage or do you pay for compute as in do you transcode on the fly like jellyfin and plex does and then you need like gpus right or do you pre-transcode and you know you serve you save multiple versions you store multiple versions and then you and then you serve those and then you have so many codecs i I mean, it's just not even funny. Like AV1, H265, like what do you pick? Different phones, different devices. It's not an easy, like MP3 at this point is like, almost like a universal format.

1:34:15There isn't a video equivalent. That's a hard problem. Yeah. Anyway. That is what makes video dramatically harder. And that's, I think, where the divide is at. And that's why I brought that up is because you got this divide of, you know, the potential of, you know, the precursor is that almost everyone that I'm aware of at least is still paying attention to podcasts via not really a podcast client anymore they're usually on some sort of platform and they're usually they're like they they ask me what show do I produce and I tell them and they immediately open up YouTube and they start searching for it now was it this one it's like yes okay so that's where folks are tending to go and here we are optimizing for this and the migration may be that do the worlds eventually collide how do they work long term etc still no and i think yeah exactly do you transcode it on the fly or do you make multi-versions of it i think you just don't do that unless you know for sure you should just transcode if you can yeah i know the jelly thing that's exactly what it does and i really like it for that because it's very good on storage but the cdn the one that i use does transcoding so they store multiple versions um now luckily i capped the maximum and i i think i only published 1080p on the cdn because of the multiple versions which they transcode and they make available but on jellyfin it's just like the 4k one so that's my that's my approach to it what are your thoughts on number four here four or five and the question mark here well changelog.com new wiring what i was thinking is there's a few few utilities that we're using that need an upgrade for example dagger needs an upgrade really badly it's like such an old version on on the changelog upgrade or replace still undecided we'll see how that goes uh the deploys i wanted to improve them for a while there was always something else so improving that time to deploy i think was like four minutes last or three minutes.

1:36:17There was two minutes at some point and went back to three minutes again. And I know that at least a minute and a half of that is fly.io. So what do we need to optimize there? So the deploys are a little bit quicker. Postgres, I mean, it's been stuck on, I think, 16 at this point, I think, 16 point something. So maybe we want to upgrade something there so we don't fall behind too much. And replacing overmind with run it. So overmind is a supervisor that runs for example the log manager um the log manager it runs varnish vinyl that runs the proxies it runs multiple things in the context of the pipe dream and pipely really but um overmine sometimes some things can get stuck because of just how it's configured there's like some some duct taping there especially when the logs how the logs are streamed so that's something that I'd like to improve.

1:37:10And I know that run it, I've used it in the past. It's very reliable. It's a very old supervisor, very Unixy supervisor. So I'd like to replace overmind, which is Go-based with run it, which is much more old school and it does everything we need. So that's like an improvement, but that's a Pipely, Pipedream improvement. And the question mark was like, what else? I think we tackle the question mark in the conversation. Gotcha. All right. So we're almost at the end. And if you like this as a listener, as a viewer, you can like, subscribe, you know the drill and comment, right? That's something that is also an option.

1:37:48I mean, we touched on many things. Maybe you have a few ideas of how to do things better, or maybe there's a few suggestions that listeners and watchers have. um i'll be more than happy to answer any follow-up questions as we will all um suggestions maybe for the next get together for the next changelog get together um you can do it on the youtube video i i think you uh do people by the way comment on youtube videos yeah a little bit the ones that you post yeah okay and do you reply to those comments oh yes that's a lot of work i just I discovered that recently last week. Oh man. Like when you get like a hundred comments, it takes a while to go through them, but it's a good problem to have for sure.

1:38:31Zulip also works. I think all of us are there or even GitHub. That's also, we have the discussion for this Kaizen. Right. And just remember to tag me because otherwise I will miss your message. So I have many things on mute. And unless you CC, I'll miss it. Absolutely. Same. Too many inbounds must get tagged or CC. one more thing last thing last thing and then we're done okay okay last thing so you know about make it work tv uh make it work dot club is something new so the hundred hundred gigabit home lab comes from make it work about the club uh it's on school it's a community of the most loyal make it work dot tv members but also those that want to go beyond just watching so the ones that want to interact we meet every two weeks both adam and jared have an invite while you were talking i send you an invite so you can join uh you can see the various conversations which are happening there and the next one is tomorrow and it's usually every other friday it's usually 9 a.m pacific time yesterday's tomorrow's one is going to be 7 a.m because some of us have kids and meetings and other commitments it's going to be before work for some people where we'll be talking about a smart garage door opener.

1:39:51We talk Home Labs. We talk Talos Linux that comes up quite a lot. Quite a few things. Kubernetes, it's all there. You can go log in and check it out. Adam and Jared, you are part of it. I just wanted to get it. I just wanted to get it into a point where there's like enough to show and enough for you to see. So it's been going on for about a month now, a month and a half. there's plenty of threads there's only i think 17 members so it's not that many it still feels like a small community has a small vibe to it um but it's a bit like this but uh with with more people so it can be a bit more chaotic i think the 100 gigabit i think were maybe nine or ten people so it was quite the group discuss but still with a you know presentation and focusing on i I mean, you've seen the video, Jared, so you know what that was like.

1:40:45Cool. But did you get your invites? Just double checking that in your email. I just want to make sure that that worked. I got mine right here. You got yours. Adam, did you get yours? Let me see if I got mine. And then you can decide whether you want to accept it or not. But I just wanted that to be out there. I do have my invite. I do see it. Yes. Thank you. I have my invite. So this is a changelog++ equivalent. you can think of it like the changelog plus plus equivalent gearhard plus plus yeah you can drop any time um when we when we meet every every two weeks and you're more than welcome to look at the threads comments ask for like someone for example misha he was asking he wants to build his own router like a router how how the americans pronounce it so a router router you go yeah so he he wants to wants to build that and nabeel built for example he's it's like a smart garage door opener he just didn't want to get out of the car to for the door to open so now he has like the all that like program so he's going to talk about it tomorrow interesting and yeah there's quite a few things so check it out yeah all right that was me how do we want to wrap it up you put a bowl in that that present there that was cool uh talking about a garage door open be kind of cool i just talked to my phone i just tell siri open the main garage and she makes it happen and it's just part of apple home kit and the fact that my garage door opener is on the network and it has those kinds of i really didn't do anything to make that happen besides just flip a switch and talk to it and that was kind of cool now sometimes she's like you don't have a main garage door and i'm like no no let's try this again hey siri do this and she's like okay gosh don't shush don't be open to my garage girl i had to stop right now yeah she was she was hearing me say her name she's she's excited she's always like kill me up in the garage please or maybe not or maybe what are you keeping there what are you keeping like breaking bad sort of situation going on yeah yeah yeah no no no it's not that yeah um so but but someone had to set up the smart garage door for you right I mean, did you set it up?

1:43:14Because you need to have the whole, it needs to be hooked up, right, to your home network. All I did was enable the Wi-Fi access to my network. And the garage door opener is on the network. It has an app that runs it. And the app allows me to install, I guess, it's been so long since I've touched it, so I don't remember how I did it. But it was like shortcuts, I guess, essentially. You can create shortcuts on your iPhone. And so that's all I did was just leverage the shortcuts that talk to the app that has the authentication to the thing via the network. And that's whether I'm at home or not at home.

1:43:53So it's not even land bound. It's it's wind bound. So it's really awesome. I can be literally in the mountains with very little service and I can tell my garage door to open or close. and i can even tell if it's open or closed because if i say hey close it she's like uh i'm closing your main garage she's like oops already done you know it's like it's she reminds me that it's already closed so i didn't really have to do much to do that thankfully but to set it up so for to take take a regular garage door right a normal that part a normal one that's what to build it a non-network one yes that would be yes that'd be dope honestly that's what he's going to talk about how he set up the whole thing like all the devices what did he pick how did he connect everything and like it was just like a regular garage door it had nothing and then he made it smart well the good thing about those garage doors they tend to have an outlet which has two two plugs one being used by the garage door opener and then one that's used for nothing basically so thankfully if you needed an outlet for your device to pair it to i'm assuming the answer is maybe yeah right we'll find out make it work mine mine is just mine mine is just a regular one exactly so that's what i want to do like if i if i wanted to set this up what would i need to do to make it work indeed yeah as a smart one i think the the thought pattern around how to tackle that you know i simplified it by just having one that was already networked but if you have one that is not networked, then there you go.

1:45:27You've got to create the network. I know how we can end this show. I can just ask Gerhard to review my new hat. Your new hat. Yes. I thought it was there. Yes. Oh, I'm so happy. What do you think? I'm so happy. That was not, I think it looks amazing. I know you like blue. I love it. I love it. That's blue. It's blue. I was looking for it everywhere. I thought I think I left it in Jared's truck. You did. It makes me so happy to know that you have it. I've got it here for you. When we get together again, I will bring it to you. There you go. It's yours. It's yours. It's mine? I'll bring you a gift.

1:45:59You give it to me? It suits you so well. I know that you look like hats. You're right. It does. Thank you. It's yours, Jared. It makes me so happy. It's your hat now, Jared. Hats on to you, Jared. Hats on. Hats off to Gerhard. Hats on to me. I think we may have a title there. I think we might. Hats off to Gerhard. All right, Kaizen. Good stuff, y 'all. Kaizen. It was awesome. Bye, friends. Oh man, our Kaizen episodes are always so much fun. You just never know what Gerhard might have up his sleeve. If you enjoy these, let us hear it in the comments. And tell your friends too. Even after 16 years of doing this, word of mouth is still the number one way people find out about the changelog.

1:46:41Thanks again to our partners at Fly.io and to our sponsors of this episode, augmentcode.com and agency.org. That's A-G-N-T-C-Y dot org. And thanks to Breakmaster Cylinder, we have the best beats in the biz. Next week on The Pod, news on Monday, Adam Jacob from Citizen Initiative. On Wednesday and on Friday, Adam and I hit you with something spooky. Have yourself a great weekend. Lips of knowledge are a precious jewel. And let's talk again real soon.

1:47:32Game on!