Kaizen! Let it crash (Friends)

49:39You can read it. 93 % of requests never touch your servers, massive cost savings on compute. Do you know how many requests per second the application is serving? Like maximum, by the way. What's the maximum RPS for this amazing Elixir Phoenix application for the homepage? Probably a lot. Gosh. Thousands? Tens of thousands? Maximum. Okay. Jared? 100 ,000? The database connection is involved. Concurrently? Concurrently, yes. I don't know. I'd say a lot. Not very many. To our homepage? I'd be like 12. 12 requests a second. Yeah. 17. 17. I'm right in there, baby. some of those those are code so 70 requests per second so if all these requests were hitting the application we need so much compute to serve that you know so much caching obviously we've we've removed all the caching now we're joking about this because we purposefully removed all the caching from the application right i remember that a couple of years back because we said this has no place in the application the application application gets restarted yeah we need to store this somewhere we need to cluster it was just just really messy to handle it at that layer which is why we introduced this five plus days running without any issues by the way this is this is like the last deploy so maybe by the next heisen if we do no more deploys we'll be able to see how it handles zero failures on the infrastructure side and three terabytes of data serve to users three terabytes so in five days this one instance served three terabytes without your application servers breaking a sweat storage is getting full so we need basically more storage for the podcast audience oh yeah it's a fun imagine a really good receptionist at a busy office this varnish server is like having someone at the front desk who remembers everything out of 100 people who walk in asking questions 93 of them get their answers immediately from the receptionist without ever bothering the experts in the back office what's cool it's been running for over five days straight without a coffee break or a single mistake that sounds cruel to me but let's go with it it served three terabytes of data that's like streaming about a thousand hd movies this one instance streamed a thousand hd movies in five days and the experts had only had to answer seven percent of the questions the one quirk the filing cabinet is getting full it's like when your receptionist's desk drawers are stuffed and they occasionally have to throw away old notes to make room for new ones not the crisis just time to get a bigger cabinet it okay i think the last the fun stats of 300 workers i think that's that's too deep that's good fun there good job do we care about gpt or gemini we can only we can only use one we can only pick one gemini's getting some good hotness let's check gemini we'll see how it adds up oh it's still thinking let's see i think it's finished maybe that's uh let me just close that um did it finish i think it did all right so uh let's go up slow thinking um i did like the thinking i could have shown pro as well show thinking show thinking show slow think i thought i said slow thinking i was like oh my god show thinking there's quite a lot there anyway we're not going to look into that so no the instance has been up for 5.3 days the mgt uptime i like it It's telling me which of those, that long list of counters is important.

53:22From a system perspective, the threading model is perfectly dialed in. 300 threads across two pools with a zero threads limited and zero thread queue length. The kernel isn't context switching unnecessarily and we aren't starting for execution units. However, we have a critical IO and storage allocation bottleneck in the SMF, the file, I'm not sure what Steve Doerr means. That's a weird one. i'm not sure gemini knows what steve norway knows either the problem is smf disk c fail is at 19 000 plus against only request at a 73 failure rate for disk allocations the g space in the g space we have only 145 megabytes while we have 48 megabytes uh 48 gigabytes used you're effectively out of contiguous blocks or hitting the ceiling of the pre-allocated file that sounds right varnish is forced to fall back to sma ram or simply not cache large objects while sma has 2.1 gigabytes free the disk is thrashing the hardware if this is spinning glass your ios is likely spinning it's not it's nvme uh so yeah there's there's quite a few things there interesting um is low relative suggesting the lock contention on the shared memory log is negligible nice you have low lock contention the bottleneck the bottleneck is strictly at the storage layer we just need basically a bigger disk the elixir developer let it crash proponent i like it your app backend is rock solid well done jared varnish hasn't seen a single back and fail or an unhealthy event in five days the let it crash philosophy is working if the app is crashing it's recovering so fast varnish doesn't even notice i don't think it is i think the application is good the good news is we're seeing an effective 75 hit rate now this is different right we had 93 there so i would like to understand the discrepancy that's another reason why you should use more than one by the way and then obviously double check the data because sometimes they give you things which aren't true the crash concern the disk cache is crashing again this is not for the application developer i think this is for the sysadmin in me i think the straight talk your main synth is high so we have health checks there's a delta responses so we have a lot of synthetic requests again sorry synthetic responses this again like a varnish thing the business person efficiency or currently serving of our traffic from varnish i think i think i know what's happened i don't think it's taking into account the synthetic requests those should be removed from the total number of requests you think claude has the right number i think so yeah yeah i think so this means for customers that's we have cost efficiency that's good the risk there's the bottom line i think this was the fun but i think this is a library i i think we can stop it here the library analogy versus the secretary analogy i think that was a better one i got a barista one i thought it was like a very good one oh yeah so you know for what for queuing yeah yeah like the barista analogy i thought was very good yeah um this is using books and whatnot the library hasn't burned out that is fun that is fun so i think jamb and i is getting a bit funnier the nightly feeds and the app are still humming along nice so that's what we have and and that was only half the problem well friends this episode is brought to you by squarespace the all-in-one platform for building your online presence with that's a portfolio, a consulting business, or finally shipping that side project landing page, you've just been meaning to do, but never get to.

57:07Here's the thing, you mass produce code on the daily. You deploy new services, new infrastructure, new hardware, you're versioning your APIs, you're simmering all over the place. But when someone asks you about your own personal website, it's like, ah, I'm still working on it. Does that sound familiar? Squarespace exists so you don't have to treat your personal site like a weekend project that never ships. Pick a template and drag and drop your way to something that actually looks good and move on with your life. No wrestling with CSS. No, I'll just build my own static site generator again. It's just done.

57:40If you do consulting or freelance work on the side, Squarespace handles the whole entire workflow. Showcase your services, let clients book time directly on your calendar, send professional invoices, and get paid online. It's the boring infrastructure that you don't want to build for yourself. And for those of you out there who are doing courses or gated content or educational stuff, tutorials, workshops, that intro to whatever series you keep talking about, you can set up a membership area with a paywall and start earning recurring revenue. Set your price, get the content, and you're done. And they've also added Blueprint AI.

58:15This generates a custom site based on your industry, your goals, your style preferences. It's not going to replace your design skills by any means, but it'll get you about 80 % of the way there in about five minutes. Here's the call to action. This is what I want you to do. Go to squarespace.com slash changelog for a free trial. And when you're ready to launch, use our offer code changelog and save 10 % off your first purchase of a website or a domain. Again, squarespace.com slash changelog.

58:48That was only half the problem. So we're like at the midpoint. I was feeling good. I feel like we had it all fixed. What else is the problem? oh wow this is like when all the fun begins so you remember this jared yes mp3 requests intermittently hang in newark new jersey this was our good friend john spurlock who's been on the show before and is a podcast nerd in fact he runs op3.dev and other podcast nerdery things and so he really knows his stuff and so when he reports issues uh you know i don't say did you try rebooting i take it seriously so i shared it with you and he actually uh did some additional digging for us go ahead so in terms of you tested this i think you had issues as well so we've confirmed this for sure i did like certain times certain files actually it'd be all requests at certain times i assume that that was that particular pop as we could call them or pipe in the pipe dream was hanging and then it would go away and he actually had the same problem he had a friday night deploy of friends and he's trying to listen to it on friday couldn't get to it by saturday morning he can get to it so it's intermittent hanging very difficult to diagnose very difficult i assume to debug and then it just comes back to normal i thought it was maybe the out of memory thing like it's just in some sort of fugue state until it reboots and then it works again but you go ahead that's what i thought that's why like a deep dive on this this was november end of november beginning of november so november i was just trying to figure out what on earth was going on just like you know like from the sides i didn't have too much time but if you look at this response there's there's quite a few things there uh this is like my initial one like an investigation trying to understand what's happening giving a couple of uh debug headers like a couple of extra headers that the request can be made of sorry can be can be um run with so we just get a bit more details forcing regions as well so there's quite a few things there i was checking into that this is uh don mckinnon um he also had issues today um so he pasted some results so thank you thank you don for adding this this was this was helpful so this is long i'm still scrolling i'm still scrolling there you go super helpful i have confirmed so i've confirmed that the requests have been hanging um you are getting the hangs this afternoon as well this was only three weeks ago so this has been going on for a while i dug deeper and i found the problem the problem was that in the fly config we had the concurrency set to connections not requests so it's possible to configure an application again you're configuring the fly proxy that's in front of the application to limit how much traffic hits your application so requests how many requests per second should the fly proxy forward to your application before it stops like that because you want you don't want to get overloaded so before it like starts throttling it starts slowing clients down and then when you start that's when you start seeing fly edge errors connections you would use for something that has long-running connections like a database for example in our case is not a database right it's an http application so requests would have been the right concurrency i have no idea why i picked connections it was the wrong one but the effect was as you can see here we had 2700 long running connections on that edge so on that region so this in this case it was i think orange one i think ewr right so ewr was getting had like all these connections opened the clients were was full no more connections could be forwarded to the application long-running connections there are usually clients which are not doing the right thing right you shouldn't have that many long-running connections so the problem was a misconfiguration on our side which meant that connections like slow connections long-running connections were basically blocking other connections from coming through so that was the problem there and i thought that was it but but there was more so uh this last comment last week we now have a check that runs every hour and what was interesting and i'll talk about the check as well we had response bodies timing out in two regions so 13 regions were fine but even after this configuration there were two regions i had an ewr where when we were using http2 and for some reason this is important when we're using HTTP2 and the proxy the fly proxy would see this it would not forward the connection correctly as in it would start it would like serve the response like we could see the headers coming back from our instances what we wouldn't get is the body so the body would always be like zero bytes served and we could see this happening we could see the connections that by the way they were opened they shouldn't have been open because the application changed so these connections should have been dropped there was something not quite right my suspicion is with a fly proxy layer because when we were forcing when we were forcing http1 everything was working fine and by the way the fly proxy when it talks to our varnish instance it's using http1 and you can see that in the headers so the proxy to the varnish was fine but the client to the proxy was not fine and http 2.0 is a very complex protocol there's so many things which just don't work the way people would expect So...

1:04:51anyway the issue fixed itself that's the thing so opening this satisfying yeah that was very nice to see and there was something uh my isle russ how would you read this uh my my illrus illrus there you go so just someone on the on the fly community forum that was very helpful they noticed that we had a misconfiguration in our fly tunnel and we were using services as well as http service and this is bad by the way this is very very bad so um everything was happy like we could push this config you know the applications were running everything was fine but because we had these two things together it was apparently creating some issues and all we did we were explicitly setting the idle timeout and the idle timeout that's the one where if after 60 seconds the connection isn't doing anything it will be forcefully terminated by the proxy so that part was important so anyway we made the change we pushed the change but even before we pushed the change the proxy started behaving and now there's pull request 49 has like we right-sized it we we made a few changes i captured like all the details the configuration the commands it's all there if you want to read it but most importantly now we have a check that runs against all regions every hour on the hour cicd it's using hurl and what i'm thinking is shall we try running that locally to see how it behaves because that's how he started it like i was starting um to do it locally so on the left hand side i'm back in the terminal on the left hand side i am monitoring my internet connection remember that christmas tree this is related to that christmas tree so i'm at the top of the christmas tree i'm at the gateway the core router it's a microtix ccr 2004 uh pretty good uh 10 gigabits per second maximum now my internet connection isn't 10 gigabits but it's 2.5 which is plenty for this test so we every second is showing me how many packets and how many bits we're receiving and transmitting okay and again we are recording everything's happening live so you can see jumping right as riverside we're pushing more data to riverside cool so So I'm going to run now just check and just check by default.

1:07:19It's one of the commands, the just command that we have in the pipely repository and check all it does. It runs hurl with a couple of flags. It downloads an MP3 file. It downloads feeds. It basically connects to all the different backends and it sees how quickly it can get data back. We're transferring about those quick, those eight seconds. i'm going to run it again as i run this pay attention the left hand side it will go to

1:07:51120 megabits per second so that's that mp3 file being downloaded so every single time this runs a full mp3 file gets downloaded alongside a few other things okay i can open the reports we're not going to look into that because we're going to run something more interesting now we do check all and what check all does it runs the same command against all the regions i'm at 2.3 gigabits per second we're downloading all the files we can see the response is coming back ewur just sped by i had sped by so all the different endpoints are returning now i'm based in london obviously the further away you are so for example this was south america those lax so a couple of instances are slower to respond and all this happens via headers you can when you connect to fly i can tell it hey i want to connect to a specific region and then that's what routes the request to that to that to that region that's cool uh and again it's all captured in that pull request and you can see what it looks like the check call one johannesburg that's usually slow and the slowest one is tokyo for me uh sydney as well can be slow so we still haven't received responses from there uh we should get that shortly you can see i'm pulling now 50 megabits 20 megabits is just slowing down and it's just the connections between now and there the last one with there goes tokyo in 60 seconds i pulled about two gigs roughly it's a lot of data that gets pulled down uh the feeds between between all of that and anyone can run it i would recommend you not to run this because we we have to pay for this bandwidth but um our ci runs it just to make sure that everything works and if we look at every hour i think i'm going to tune this down you can see there were no more connections hanging so we got to the bottom of that as well if it ever comes back because it went way on its own it comes back on its own we're all about it exactly now we have a system that is able to inform us when there's a problem so let's go to three we're on page number three this one for example took more than five minutes right so sometimes when the connectivity is a bit slow some some regions can be slow that's when you get these timeouts so this is capped at five minutes the last one that failed was a while ago so you can see we're january 5th there we go there's one that failed january 4th check all instances so let's see run and we'll see exactly which region failed execution nrt that's tokyo and as you can see we have a hundred seconds right so if after 100 seconds it doesn't download it just just times out and we were pulling data so it but it didn't finish downloading the entire mp3 and we're downloading a hundred and something megabytes very cool so i mean not cool that it didn't finish but cool that that was a while ago and we can actually test this now does it do we need to be doing such a large file is that part of the the test or can we test a smaller file and still get the same results we could yes this was a file that was reported so we need to find an mp3 file absolutely i think we can also reduce the frequency um we don't have to run it every hour this was always in preparation for this conversation about episode 456 that's coming up that's coming up that that's that's the deepest rabbit hole so i'm leaving that that's i'm leaving that for last that's coming adam one thing i suggested though in in our uh the nirzula but i don't think this is i didn't check to see if this is even a thing, but to validate, you know, if the fly CLI could validate the toml file for you, because you could have been, you could have checked the toml file for syntax errors or just do's and don'ts, essentially.

1:11:37And it didn't. It does have a validation subcommand. Syntactically, it's correct. The config is valid. I mean, it was applied, but because it combines two things, it shouldn't. So at least I would expect a warning like, hey, you're using both the HTTP services. Yeah. Validate syntax and validate, you know, expected, you know, true Tom will file config, you know, don't combine or conflate two values or overwrite one or, you know, just that kind of thing. That's how I would defensively do something like that in a CLI to protect my user from a poor config. They could have just not been holding wrong for so long.

1:12:16Yep. I agree. So it's the impact of that configuration indeed. Yep. so this is something we can see again the same logs um we can see this one here we go like to 50 megabytes per second that's 500 megabits when we have these peaks when we see this in the fly config we can see this when usually when the when the benchmarks run or like when the checks run because they put significant pressure on the instances we can see them and we can pick them up straight away so that's that's what this is all right so remember this guy this guy was saying march 29th so it's almost two years ago when this guy was saying we will run into all sorts of issues that we end up sinking all kinds of time into so this guy had a good hunch this is jared march 29th and we just just went through a couple of examples of issues that we had to deal with part of this but because of this we understand the traffic and we understand how the application behaves and the backends behave at a very deep level so you're right jared we did sunk all sorts of how many lines let's see how many lines do we have now so how many lines 20 lines 590 lines 590 lines we have in total of varnish config it's more than 20 lines by the way we have like the roadmap to 2.0 this is 1.0 that we tagged and shipped it solved like a lot of issues uh but that was the easy stuff okay so for everyone that stuck with us something really good is coming up and uh adam was already mentioning it episode 456 there's something special about episode 456 so what is special about it what what stands out to you jared oh it's just getting rocked with downloads so episode 4456 oh off it's complicated was down by the way this was recorded uh in 2021 it was published again august 2021 for some reason it's been downloaded a lot in recent months it has over 1 million downloads this is the most popular episode on the changelog ever the most downloaded episode it's crazy it's crazy so oh so you guys looked into this we did yes we we dug into this okay i didn't know you guys were doing this so we just had a quick look to understand what is happening here um so we have honeycomb open up remember every single request which comes through the pipe dream through pipely every single request we sent to honeycomb we're able to uh look at it This is the last 60 days and I have filtering done in such a way so that I'm only looking at this one file.

1:15:07How many times has this file been downloaded in the last two months? And you can see the peaks, right? You can see, and by the way, this is gigabytes. So, and this is the period is four hours. So we are peaking at about a hundred, well, actually it was like this peak was here. we had 200 almost 300 300 400 anyway close to 400 gigabytes in a four-hour period it's just too much i think so like i know this i know this is a great episode great conversation but i remember that conversation it was good like who is downloading this file 400 i know times or actually more than 400 times every four hours consistently for months on end and super fan super fan so we can see like a different regions now this is spread across the entire world it's not just one region this is really really big i think if there was a ddos attack i think this would class this one and uh like in the last six months in sorry in the last two months 60 days we served 30 terabytes in san jose california alone in tokyo we served 5 15 terabytes this is this is a big number and if you look in this column the distinct ips the client ips we had over 10 000 ips downloading this file so this is not one or two ips this is thousands and thousands of IPs which keep downloading this file over and over and over again so I don't know how we would block about 10 ,000 IPs right that would be that would be the VCL would be crazy well that episode was starring Aaron Parecki who is a very talented person and he is the co-founder of Indie Web Camp and a big fan of the Indie Web as well as OAuth obviously so my hunch is Aaron's very interested in being the most downloaded episode ever and he and he controls a fleet of machines from all around the world and he points them wherever he wishes and he thinks you know i'm gonna do i'm gonna get the number one spot on these guys download charts and so i'm thinking aaron parecki is you know the man with the mask on we pull a mask off and it's him this whole time what do you think you heard i i think that we need to speak um see i don't want to say the specific language i think we need to go to asia i think we need to visit a couple of cities in asia okay find the ips which are responsible for this because this is a crazy amount of traffic asia it just so happens if we look at so asia is the basically the continent which where we are getting the most downloads from because of this one episode and this is actually traffic being served this is not like head requests or get requests these are bytes being sent to thousands and thousands of machines in asia every single hour so whoever is doing this please stop please so we need to like knock on doors we like go over there and knock on some doors and say excuse me do you are this is this ip address at this home and then they might say yes and say would you please stop and what's going on over here what can they possibly benefit from this like what could they be getting maybe maybe we're the speed test someone is using us to speed test their connection who knows yeah maybe that's the only thing i i can imagine but that's a lot of ip addresses it is and it's across multiple regions which multiple data centers yes so multiple regional regions fly regions are serving these ips yes they're all coming from asia by the way again i don't want to mention any names because there's there's there's no no one there's no bad guys here right we just want to assume that someone left the oven on i don't know man it's like the blinker on when you're driving i was like hey you're not turning it's it's time to turn that blinker off the so the way the way i can see us mitigating this and this is a hard problem because of the number right of ips which are hitting is we can basically start blocking entire net blocks entire entire network blocks unfortunately some genuine listeners might be caught in this and basically a changelog will not be available or at least the mp3s will not be available to a portion of users the other one is obviously we can we and we should right this like the next problem we should enable some throttling because there's more stuff happening here so we don't have any sort of throttling we assume fairness we're assuming goodwill we're assuming decency and we're not seeing that here that's the internet so to be honest like whoever is doing this and it's not llm so i had look we have we have that problem as well but in this case it's not llm this is something completely different so my hope is by someone that listens to this episode maybe we put this in the intro whoever's downloading episode four five six please stop because we'll need to take the next step.

1:20:14I know it's a bit of a cat and a mouse game, but that's what will need to happen because we need to pay for this bandwidth. This is only Varnish, right? This is only the cache layer this is happening. This is only the cache layer, yes. And so what mechanisms are in Varnish to do throttling or rate limiting or just anything like that whatsoever? There's VMODs, which are basically modules that Varnish loads that just give it extra functionality. One such Vmode, and I've looked at this, it is free and open source is the vmothrottle now that means that we need to start keeping track of ips and it will use a bit more memory that's okay we have more memory and then we can need to start basically applying limits to how many download specific ips can do and we can limit it to mp3 files only so if we have a bot or if we have for example like i don't know an rss aggregator or something like that we can we we were okay serving those requests because again that's what varnish is meant to do the problem here is that we're serving a lot of bytes for mp3s the same mp3 that cannot be real traffic yeah i mean even in this case you can like tie it potentially just to this mp3 like you just said which is not an all mp3 scenario like if you request this mp3 with this kind of like request signature of x per whatever i mean i didn't i didn't examine the actual signature of the request but that's how i'd probably investigate it is begin to isolate yeah does that require us to write a lot of defensive code against that kind of scenario i don't think so it's just some configuration we just need to add more configuration and back to jared's point we're just chasing now new problems that we didn't even think we would have um but we have what looks to me like an actor that's not very um i want to say this in a nice way an unfriendly actor that is not very happy and they you know are very angrily downloading our mp3 over and over and over again thousands of times across thousands of ips and this is not cool because ultimately we end up paying for this bandwidth that is not helping anyone but that's one it's not the only one so we have one more so you can see here for example um this is the last last seven days we have seven terabytes that were transferred in the last seven days seven terabytes maybe that's more than that it needs to be more and it actually geoco does not exist okay i was expecting to see more than that anyway asia is the one that we can to see like like that pattern but we also have like in europe sometimes we have these spikes and it's like this spike which i wanted to focus on we know that someone in frankfurt that connects to frankfurt downloaded the static favicon 170 000 times in the span of i know like an hour or two so they downloaded this like two three hours so it gets you know requests like this that are putting stress on these instances i mean what potentially that was like a pass request as well which means it like went through the cache which means that they must have had like a cookie setup or something like that that basically was preventing the cache from working in this case which again that's how it's supposed to work so anyway i think i think that was unfortunately not the best thing that we could have ended on but uh it's a thing and it's something food for thought like more work to be done there's many things that we didn't get to talk to we didn't have time for for example we didn't talk about the nightly by the way nightly now is being served by by the pipe dream as well and the reason why we had to do this because that sometimes will get scraped would get hit really heavily it's a very small app it's nginx but if i open it so let's just click on that one and that's pull request 46 before it was basically topping up at 141 requests per second now it's 1300 so it's almost like a 10x in order of magnitude faster the latency went way way down so and it's just the only thing we had to do is basically put varnish in front of it nice well that's nice yeah that's one more thing there and you can go and have a look how it works.

1:24:34It'd be like a benchmark here, a small benchmark here. That's it. We have, we have last one for the road, but before we do that, anything else that we want to talk about before I share one last thought? I suppose what we do, you know, if we know these downloads are happening, we're here on the podcast, just politely asking to stop. We just let it keep happening. Well, we could, we could set up some sort of throttling. I think it'll be the easiest thing. Now it will impact everyone. I don't want to start blocking, again, IP ranges, net blocks, because we don't know who's going to be caught there.

1:25:09They may change to other IP blocks. So that's entirely possible. We don't know how this will work. We can't block an entire country, an entire continent, especially if it's a big one. I don't think that's reasonable. So really throttling is, I think, the fairest thing. And then we can throttle MP3 specifically. because we do have for example i i see them like for example we have an hd a python client and a go client that every week they come and they download all our mp3s i don't know why they do that but every seven days they basically request every single mp3 that we have so they're like scraping the website and then pulling everything down i don't know why yeah again the closer like the more i was looking at and again because i was working so deep in this i started noticing like these behaviors that you would normally not see so it's one of the advantages i suppose to being to working so close with the traffic with all the requests and having this level of understanding and visibility into every single request so it really helps down to the ip level something like that though like the go client and the python client where would you would that be a honeycomb thing where would that be yeah it's honeycomb yeah you can filter by user agent for example and you can see that like there'll be on for example say um now i don't want to show any ips or anything like that so that's why i'm not going to screen share that yeah but once we start digging into that you can say group by client agent and you can say filter by mp3s so like url contains mp3 and that will be able to group and you can say oh and by the way only show me where there's more than, for example, 100 downloads.

1:26:54And then you'll start seeing like the outliers, which are the clients that are downloading certain MP3s or MP3s in general excessively. Now that can be spoofed. That's the other thing. Like we have, for example, the request agent, like the user agent, it's empty. It's an empty string that also happens, right? Because you don't have to send the header if you don't want to. Yeah, you can also send whatever you want to. So that can be spoofed. Yeah. And it's like whenever you build systems like this, and then even when you observe them, I guess you don't expect, that's what I originally thought, but you kind of hope that clients, aka people, behave.

1:27:35You know, they're going to use the system for the system's purpose, not to once a week download and scrape the entire thing. And I mean, in that case, somebody could have their own web archiver and they could have altruistic reasons for it. I think that's kind of silly, but you know, once per week, download the entire contents and somebody is disc seems like, uh, I want your thing and I want to keep getting your thing. And if it ever changes, I want to make sure I have that snapshot. I don't understand it. It doesn't make any sense. Like what would make anybody do that? What is the purpose and motivation to keep doing that, to even commit the compute or the script or the time to do that?

1:28:16Like what are they getting from it? I don't know. We need to go over there and knock on some doors, man. I'm going to ask them. Yeah. Why are you doing this? Every door in Asia, do you listen to the changelog? Yes, I do. How many times? Yeah. Tell us about 456. You know what 456 means, don't you? Yeah. It's just, see, this is how, so this is, I think, a really delicate and a really important point to discuss. because this is how good systems become bad systems it's true yeah you have to treat everybody bad exactly like we don't want to be doing this but we are forced to do something against something which isn't good so um it's not benefiting anyone and we have to step in and do something about it now we have to do it it's been like i was expecting this to stop but it's still even to this day we made varnish i mean now now that it's stable it's able to serve more traffic it's able to like we just had like the biggest spike because now the system is more stable but it means that bad actors again i shouldn't be using that unhappy people unhappy clients use it unhappy clients yeah the only person you can offend is the one who's doing this and i'm fine with it yeah they need to knock it off here's a cut this might be a cudgel but if we're trying to solve the problem of they're taking our bandwidth for something that's no longer relevant or interesting and it's been out there for years what if we could just toggle certain episodes and this might be a cat and mouse game as well but like at a certain point it's like well just give them the r2 url and not the cdn url and just let Cloudflare deal with it.

1:30:02You know, like just let them download it directly from Cloudflare. And we're just out of the equation then. We don't care about the stats. We don't care about anything. We're just like, you know, we've served this file plenty of times through our CDN. Now we're going to just let R2 serve it. What do you think about that idea? I can see for this specific episode being a very simple fix, right? Because we can just serve basically a location header and we just do redirect and that's it. We're done with it. So it'd be like another synthetic response. The question is if they're actually malicious, then they switch to a new episode and start doing that one.

1:30:35Exactly. Exactly. And we have other clients, which are, for example, we've seen that pass, right? They're basically busting the cache and then purposefully going to R2 directly and just varnishes like a, almost like acts like a proxy in this case. Right. So we have that as well. We have every now and then we have like this, this random client that comes and downloads all the episodes. And that's not the problem. So I think that some sort of a throttle would make sense, which would keep the system fair to everybody. But the throttle will need to be high enough so that it doesn't impact anyone else.

1:31:09Now, if, for example, our requests or like if our audience grows or we become more popular and we get more requests, obviously we would need to be aware of where the limit is and start increasing the limits, right? Once we are throttling too much, maybe. that seems to be more like long term and it seems a more i don't like a well-engineered approach in a way but certainly the simplest thing would be just like take this one url i mean that could be done in minutes or all it out and then we would stop this abuse for this specific mp3 that would be the easiest thing for sure um so yeah i can see the how pragmatic that approach is and i like the pragmatism well it's at least worth checking to see if you know the mouse is still alive over there right you know yep and if they are well then we'll know that this is a cat and mouse game but if it's just like somebody left the blinker on we're just going to turn their blinker off for them yeah and see if it just the problem goes away and if it changes to a new mp3 then yeah we need more generic solutions yeah we may not need that at all i do have to say that the internet these days is very different from the internet even like a year ago with the rise of llms and ais i'm starting to see patterns in our traffic which are unlike any other time we have these very big spikes when a lot of data is being requested in very short periods of time from i mean the user agents they don't make much sense i mean i know they're spoofed there are many ips which are being used so it's almost like there's like a i know some for some system which wants a lot of our content is seeing silly things because some requests they just don't make sense um like for example what benefit does static favicon have like what's up with that that just makes no sense it's a small file maybe it's a heartbeat or a version of a heartbeat maybe but this is the first time i've seen i've seen this specific file being downloaded this many times i haven't seen this before which makes me think is this a trend that we'll start seeing more and more requests that don't make sense and then you start having to set up like some form of protection for all sorts of clients that are just doing the wrong thing you need like a defensive layer by default exactly exactly yeah and something that would be fair to regular clients like for example when i want to do like a benchmark i mean sure it's me and i wouldn't want other people to do that because i'm testing the system making sure like the real world the production system everywhere in the world is working correctly and i'm aware of what that means and how it costs and by the way my ips are removed from all like the the stats because otherwise you see like those massive benchmarks so we account for that but we can't account can't account for all these like weird clients it's a challenge i think it's a good one but it just sets us up to you know when you become older it feels like this like a more like often like an adult problem right so we like got the thing barely working we got it out there we know we made it stable reliable all that now we're hitting almost like feels like a like a new layer of problems and then this to me is like a hint as to like the next phase oh to be a kid again yeah well right one positive thing i think is the robustness of our observability like being able to have this visibility is great because otherwise we're like you know wow pat ourselves on the back Aaron Parecki let's get you back on the pod because man you are big all over the world that's amazing they can download you know so what's your one last thing for the road ahead I agree what's my one last thing yeah so keep it short we'll keep it fun I mentioned about the Christmas tree I mentioned about the various things which i had going um over the holidays so make it work club that's the place you're there both of you are there so you can join whenever you want next thursday yeah next thursday i'm going to talk about the 100 gigabit wan the 100 gigabit wan so why would i need such a thing smoking it's smoking for sure so i thought the ccr 2004 has like four cpus has like multiple 10 gigabit sfp plus ports it even has 24 sfp 20 sorry sfp 28 ports but it doesn't have a switch chip and people that know about a little bit about hardware you want to switch chip to the hardware offloading l3 and even l4 so after i bought the ccr 2004 it was like almost like a christmas present i thought surely this will be enough for the rest of my life and no i had to get the flagship so i'll be talking about that um the land the setup quite a few things coming up and um it just goes to show how much i enjoy the hardware side of things as well the networking side of things like i shaved two milliseconds of my wad it's amazing like like little things like like that you know like i it was already good it was like already like sub five milliseconds but i wanted like sub three milliseconds it's now 2.4 milliseconds so the next and what it means like why would i do this so first of all i'm all about improving and every winter i improve the network in this specific instance i wanted the pages just to be snappier like things to load a lot quicker to handle a bit more traffic but also to not have an impact like i was running that benchmark like 2.5 look i'm going to do another one right now let's see i have a speed test i have Speedtest right here.

1:37:04Speedtest London. Let's go for this one. So we're recording, we're streaming, right? And I'm just pulling 2.5, 2.6 gigabits down. And there's no interruption on my network, right? So it's just my bread and butter. You know, that's how I work. And by the way, if you see any buffering or any slowing down, let me know. I see Adam a bit more pixelated. Maybe you can see me pixelated too. I don't know. But yeah, I just pulled six gigabytes. three down three up and it's just just what i do every day which i work with this stuff and yeah i enjoy it and by the way this is this is a the slower gateway router so i'm getting the proper one set up and i'll talk about that um and there's so many things there like vlanning is quite a thing i have a new ipv4 block by the way so some would say that i'm preparing for hosting something and maybe i am i don't know we'll see how that how that works but i just realized that my home connection obviously i couldn't serve all the mp3s that were being downloaded like that would really cripple my connection if that was happening but i'm at 2.5 gigabits the next one will be 5 gigabits and the hardware can do it and the 5 gigabits i mean that's like a decent server and if you can do five gigabits all day every day sorry um yeah gigabits yeah gigabits per second that's pretty decent so i'm just waiting for more internet i'm gonna say you're gonna have 100 gigabit wan but you're not gonna you're not gonna have a connection for it right right so very few places in the world have that so if i was in switzerland i would get 25 gigabits now which is move would you move for this of course the only reason the only reason to move meditation yeah it's the 25 gigabit connection but i know that 100 gig is coming so we'll see are they either ship it by the time i move or i come i move and then they ship it so it's one or the other okay the important thing is i have the router to handle you'll be ready you will be ready exactly so i'm a prepper i'm prepping for that prepping for a good internet and this is just like uh and interestingly five years ago when i got like the the previous uh router i did the same thing it's like a forum post it's like a follow-up um so i just did a follow-up recently at this milestone so i've been at this for some number of years and i like optimizing my network and making sure that it's in tip-top condition relentless i love it so relentless good stuff gear hard well that's a happy note to end on right that's a happy note to end on observability in 100 gigabit no that's where you go all right well the good news for kaizen is we have a lot to work on always always that's what it seems yeah we know how to we know how to pick them don't we oh my gosh the rabbit hole goes deep and we keep going in kaizen my friends kaizen guys all right kaisen 22 is in the bag join the discussion in our community zulip head to changelog.com slash community to sign up for zero dollars and of course check out all of gerhard's passions at makeitwork.club thanks again to our partners at fly.io and to our beat freaking residents breakmaster cylinder next week on the pod news on monday damian tanner from layer code on Wednesday, and Techno Tim catches Adam up with the state of Homelab Tech on Friday.

1:40:40Have a great weekend, recommend us to a friend if you like the show, and let's talk again real soon.

1:41:04Game on.