Kaizen! Tip of the Pipely (Friends)

0:14Welcome to changelog and friends, a weekly talk show about scaling fly machines. Speaking of Fly, thanks to our awesome partners, the public cloud built for developers who ship. Learn all about it at fly.io. Okay, let's Kaizen.

0:41Well, friends, it's all about faster builds. Teams with faster builds ship faster and win over the competition. It's just science. And I'm here with Kyle Galbraith, co-founder and CEO of Depot. Okay, so Kyle, based on the premise that most teams want faster builds, that's probably a truth. If they're using CI providers with their stock configuration or GitHub actions, are they wrong? Are they not getting the fastest builds possible? I would take it a step further and say if you're using any CI provider with just the basic things that they give you, which is if you think about a CI provider, it is in essence a lowest common denominator generic VM.

1:21And then you're left to your own devices to essentially configure that VM and configure your build pipeline, effectively pushing down to you, the developer, the responsibility of optimizing and making those builds fast, making them fast, making them secure, making them cost effective, like all pushed down to you. The problem with modern day CI providers is there's still a set of features and a set of capabilities that a CI provider could give a developer that makes their builds more performant out of the box, makes their builds more cost effective out of the box and more secure out of the box.

1:58I think a lot of folks adopt GitHub Actions for its ease of implementation and being close to where their source code already lives inside of GitHub. And they do care about build performance and they do put in the work to optimize those builds. But fundamentally, CI providers today don't prioritize performance. Performance is not a top level entity inside of generic CI providers. Yes. Okay, friends, save your time, get faster builds with Depot, Docker builds, faster GitHub action runners, and distributed remote caching for Bazel, Go, Gradle, Turbo Repo, and more. Depot is on a mission to give you back your dev time and help you get faster build times with a one line code change.

2:38Learn more at depot.dev. Get started with a seven-day free trial no credit card required again depot.dev

2:51well today is a very good day because we are kaizenine and gerhard is here and adam is here and i am here hey guys hey hey it's good to be almost weren't all here but we're all here happen rain and thunder and lightning internet outages so what happened to my internet yeah to your internet i don't know i just went down and stayed down about 12 30 on monday maybe one 1 30 and i uh called them and told them my internet was down and then they said we'll fix it and then they didn't fix it and then they did fix it but it's a little bit too late for us we actually we're gonna record at 9 a.m i think on tuesday and it came back up around 11 a.m on tuesday so not even a 24 hour outage yet still way too long way too long for my liking and it was just my house i don't know what happened they said they had to rebuild the modem which was a apparently a remote rebuild i think they just flashed it with a new something or another you only have one let me guess you only have one internet this is correct well i do have my phone i said i told you guys i could just tether to my phone and and you know play hot hot and loose no i'm not saying fast fast and loose thank you i don't know i was thinking fly close to the sun and i was thinking fast and loose and i said hot and loose but gerard you said you had like a multimedia presentation i'm gonna have to like have really good internet and so we just called it off so now we're here the internet's back the rain is over i'm assuming it's done raining there adam you're all clear yeah i think so no yes we're good okay and gerhard brought us some goodies you got a story to tell tell i do yes tell us what you have to say i was thinking about this for some time actually wow and i was thinking is when we get close to launching the pipe dream to launching pipely how do i want to do this and that's the story the story is that you were thinking about it or that you've thought about it and you're going to tell us more well the story is that i will tell you more a lot of stuff has happened i decided to double down on the pipe dream on pipely i decided to like all my time went there yeah and all that means is that we have something we have something good we have something the launch story is that you're gonna i think i think it's good like let's let's just like you know let me set the expectations at the right level okay and let's see if adam approves of the toy that we want to wrap for him okay who has the bar does adam enjoy this toy that is being built in the factory and let's see what he thinks of it i love toys don't we all don't we all well let's get right to it show us a toy garard all right so i'm going to share my entire screen that's why i when i mentioned that this is a presentation style it is a presentation style i could even share the slides now they're about 80 megabytes big wow because there are some some some uh recordings as well right some multimedia something if something will not work live that's okay i already did it it's recorded the screen behind me will be out of sync with the things that we'll be doing but but the thing itself um has been captured so that we can tell a good story all right so the thing which i would like us to do now is click around a few versions of the changelog site and talk about how responsive the different versions of the changelog site feels to us And I think this is why Jared's internet was important so that, you know, he experienced it as close as he would normally.

6:45Right. No tethering, nothing like that. Right. So we will start with the origin. The origin, as our listeners will know, runs on fly. And we always capture a date when this was created. This particular origin was created in January of 2024, 12th of January. which means that the url to go to the origin by the way most users will not do that this is for the cdn to do that right is changelog dash 2024 dash zero one dash 12.fly.dev and i want you to be logged out that is important um or just use simply a private window whichever is whichever is easier we don't want any cookies we don't want you to be logged in because the experience will differ if you are and this is going to be our baseline so it's important that the reading is accurate and this is as slow as it gets right this is if we were to hit the website and this is running in ashburn virginia and it basically comes down to um your network latency to ashburn okay so let's open up the website i'm going to open it up as well and i'm going to click around and see how it feels.

8:00How responsive does it feel? That's what we're aiming for. Remember to be signed out. That's the important part. I'm clicking around. I'm signed out. I'm in a private. Perfect. Window. So how does it feel in terms of responsiveness, the website? Average. Average. What about when you click on news? Do you see any delays? Anything like that? I would say there's a slight delay. It doesn't feel as snappy. It feels like it's rendering. I can see it rendering. do images play a game into this like oh yes okay because that's what i'm noticing most that's uh laggy is like the the viewport kind of gets painted and then it moves around because the images catch up and yeah it doesn't feel it feels like it's uh like i'm on tethered internet basically right there you go so imagine if jared had a tethered internet how slow that would feel double tether cool yeah exactly something like that yeah so now the interesting thing is that even though the images do serve from the cdn everything else around them the javascript the css all of that um i don't think it does let me just double check that oh it should it should yes actually does so it is just a request to the website you're right actually yes everything all the static assets are served from the cdn it's just a request to the website which makes it feel slow and i don't think we're biased i don't think we are imagining this i have been looking at this for quite a while and it all comes down to that initial request anything that hits the website for me takes about 360 milliseconds and this is this is constant so i'm showing here the htp stat output um a tool we talked about it we may drop a link in the show notes and that's what it comes it the slower it will get but you in the us i would have expected this to be snappier so interesting that it isn't i mean it's borderline snappy i can feel it a little bit but it's not bad right um and i think that is because you have the changelog.com experience so now if you go to changelog.com and exactly the same stuff that you did before changelog.com and you click around how does it feel now instant yeah i mean it's snappy versions of instant almost instant like some pages feel i think the news is the one that you notice that like that paint just just takes like a little bit longer right it's not instant doesn't load instantly but it's it's significantly better than if you were to go to the origin agreed and this will be this will be consistent for everyone i think that is that is the advantage of changelove.com actually running through the cdn all the requests run through the CDN, even the ones to the website.

10:45So the thing is that if it's not in the cache, if it's a cache miss, for me, it loads, the homepage loads in about 300 milliseconds, which is slightly better than when I go to the origin, but it's not great. Now, obviously, if this is a cache hit, in my case, it loads in under 20 milliseconds or around 20 milliseconds. And 15 times quicker is a noticeable difference sure so as soon as these things get cached it's really really fast so we would expect this from a cdn all the time but it should consistently behave like this and by the way title proposal 15x quicker maybe we'll see we'll see we'll see right we're getting there note taken so the problem is that with a current cdn 75 percent of home page requests are cash misses so 75 percent of which is to me insane it is insane right that sounds pretty bad so some would say present company included okay it defeats the purpose of a cdn right i would agree yeah but there's more but there's so there's more tell us tell us so here this is a question for you both adam and jared what do you think is the percentage of all get application requests that our cache hits how many of all the requests that go to the app to the origin do you think are being served from the cash and the options are 15 20 25 or 30 what is your guess well i buzzed didn't think it was a game show my bad uh i buzzed myself in even go ahead you gotta go first 20 please 20 okay jared so you just told us that 75 are misses yep and that's every type of request now you're asking no no sorry 70 just the home page just the home page is 75 miss which means the home page is a 25 hit now i'm asking about all the requests requests to the application origin remember we have a few origins okay just gonna be just the application i'm going with the highest possible choice 30 so yeah 17.93 so yes adam is closer 15 would have been accurate uh 20 like i think 20 is more accurate because 8 17.93 is closer to 20 sure so yeah i think uh you were too optimistic because if 30 were cash hits that would be good it's actually 17 18 18 18 are cash hits everything else is a miss and the window is the last seven days the last seven days so in the last seven days only 18 of requests were served from a cash how does this make any sense right so october 2023 this is this is what we started right on this journey when uh this was issue 486 in our repo what is the problem well after october 8th 2023 cdn cash miss is increased by 7x it just happened we looked into it we tried to understand it and we could not and it's been ever since then or it's like this systematic problem ever since then Well, it has been low ever since.

14:24So the cache hits have been low to the application specifically ever since, which is why even when you go through the CDN and you think, right, things are snappier and they are to some extent, many requests, they are just cache misses, especially going to the application. So here we are today. It's only been three weeks. so it's only been three weeks so let me explain what it means so many depends how you count okay okay so uh the thing is that um roughly that's that's how much time i had to spend on this like about three weeks in total right spread over like a long period of time right so we are just about to unleash our clicks on the pipedream.changelog.com.

15:16Bring your mice out and let's do this. Let's unleash our clicks. Pipedream. And by the way, anyone can reproduce the same experiment. Remember to be logged out. That part is important. Or a private window, because if you have any cookies, it will bypass the CDN. That's the rule. When should I do this? Right now? Right now. Oh, yeah. Right now. Just click around and tell me how it feels. I mean, I've tested it myself, but I don't have your experience. So how does it behave on your side of the world? So one thing in particular that I notice between the two of them right away, because I clicked into news and it seems like there's this paint delay on the right hand side.

16:02So we split that viewport and news. Left side is subscribe. Right side is the newsletter. Very, very cool. But that right side newsletter side, the background color seems to like delay paint. I'm not sure if that's happening here as well as the past. That's an iframe. So that's a secondary request. I got you. Okay, so I'm not going to judge that then. I think that's important. I mean, it's fair to judge it. It's like the whole thing. Like, how does one compare to the other? Where's the iFraming from, though? iFraming from? From the same site. The same site, yeah. It should be, yeah. So for me, again, for me, when I click on use, I can see that the iFram, you're either, there's a little bit of a delay.

16:40But when it paints, for me, it paints instantly on Pipedream. On changel.com, there's like a little delay between the whole thing. that's at least like how i experience it now anyone can reproduce this and we wonder or i wonder how do you perceive the two wherever you are in the world if you click around these are live links by the way changelog.com and pipedream.changelog.com they should both behave um sorry they will both have the same content and what i'm wondering is how do you perceive them is is there a significant difference is it the same right what do you notice what about you jared do you notice anything different my experience specifically on the episode page which i think is a good one because it has a lot of let's just call it first party content not even cdn content because i do i mean the cdn is the cdn right so i do see the images lazy loading in slightly just like they would on the previous one however the first party content for instance i'm on making dn simple podcast 637 seven which has all the podcast information all the chapters and then the entire transcript which is lengthy and it loaded in very quickly um obviously my browser is not rendering that text that's off the screen but it has to at least download it in the html so that was very fast other than that it feels similar to change all.com and it's the images that i do notice load in because they're lazy loaded they load in you know a split second later other than that but yeah i think the episode page is a good test and it's significantly faster okay so pipedream.changelove.com if you look at the requests to see the network requests in your developer tools you will see that all the static assets they load from cdn2.changelove.com which okay is the pipedream 2 So everything that we serve, all the origins, whether it's the assets, whether it's the feeds or the website, it all goes through the pipe dream.

18:46And the application was changed. That's what we were talking about earlier. We may unpack that. The change is to every public URL that we serve. Now we have an alternative, which is all running through the pipe dream. i'm using an http stat here and i'm going to https pipedream.changelog.com if it's a cache hit it loads for me in 25 milliseconds which is slower than changelog.com it's five milliseconds slower in real terms 25 roughly roughly 25 slower however if it's stale it should also return within 25 milliseconds which is what's happening here our content should always be served from the cdn regardless if it's fresh or not and in this case what we see if it's already been served once it will stay in the in the cache until there's pressure on the cache and we control when that is we just basically size the cache accordingly we give it more memory and then more objects will store will remain in memory and what we want to do is to always serve content from the cdn whether it's stale or not so this was a cache hit right you can see there's a cache status header it was served from the edge we see what what region it was served from by the way if you were to do a curl request you'd see the headers you would see like all this information even in your browser developer tools open any endpoint and you get this information for every single response.

20:28We see what was the origin that the CDN had to go through to fulfill the request. The TTL, that is the important flag, which is, sorry, the important value, which is how long was that object stored in the cache? In this case, it's minus four. It's a negative number, which means that it's considered stale. The default value, the default TTL is set to 60 seconds. anything that was served 60 seconds that within 60 seconds that's sorry anything that was requested within within 60 seconds is considered fresh but then we have this other period this other value which is grace which says for for 24 hours continue serving this object from the cdn but try and fetch it from the background and also we see that this has been served from the cdn 26 times already as i read these headers these are important every single request now has them and we can see which was a region was an edge region we don't have an origin yet but we should by the way the closer you are to the origin it just says the origin shield all that we can configure now what a shield origin does basically the cdn instances which aren't close to the origin they will go to the to the to the cdn instance which is closest to the origin and that's so that we place as little load on the origin as possible i don't think that will be a problem for us but we can do it if you want to and the question is after all these years are we holding fly.io right what does that mean well changelog the application has only been deployed in in two regions right actually one region and we have two instances but we always wanted to have it spread across the world the problem with that is how do we connect to the database then you're introducing latency of the database layer but now these cd instances they can be spread around the world that finally we're doing this right right we just put it in front of our app instead of making our app be yeah distributed now we're distributing in front of it i think so yeah so shall we see where these instances are running yeah i'll see it man i'm curious are we are we curious about anything else before we move on to that i'm curious about the the rollout of this thing because because I've noticed a few things this week and I'm wondering if maybe things are pointing at different directions and, and, uh, if that explains some stuff that I've been seeing, but we can maybe hold that for later.

23:08I think we can talk about that now, just, just like, so that's where we go, we go through this. I, we never had this situation before, by the way, where we have two application instances completely separate that are pointing to the same database, right? So the data is always the same, but one is going to become the new production and it's configured in a certain way with a new cdn and the existing application the one that's behind changelog.com is still consumed by our production cdn i mean we have two cdns that that's that's the situation right and we can't change the production application because if we do that then we have rolled out the new cdn and we don't know whether we are ready yet i think that's that's what that's what we need to determine today what what else is left how do things look so far and just like assess the readiness of the new cdn yeah of the pipe dream so what things have you noticed jared that are off so i shipped changelog news monday afternoon and that particular episode has dramatically lowered downloads so low in fact that it has to be a bug somewhere in this in the system that's not real like it's not a real number or and i'm wondering if maybe a bunch of podcast apps got pointed to the new cdn and we're not capturing those logs which is how we get the stats so that that was the first thing i was like there's no way that this is actually only been downloaded 700 times or whatever it was yeah in the first day that was the first thing i noticed there and you're nodding along so you're thinking probably that's the case yeah i think so i think that that's what happened if if so depending on which instance picked up the job right like this is all like background jobs right it must have pushed a different url than the live one so then all those podcasting platforms like how would you call them all the podcast clients i mean okay so all the podcasting clients some of them maybe all of them may have picked but i think if it would have been all of them would And we'd have seen zero downloads.

25:15Yeah, it wasn't all of them. It was just some of them. And maybe eventually the other app caught up and started doing things because we sent out a bunch of notifications, you know, in the background. Now, because we have multiple instances, and I think this must be a job queue, right? Whichever instance picks up the job, basically it puts its own URL and then ships it to the actual sub-beens that we are in production without wanting. Damn. Yeah. Okay. So, I mean, assuming that all those clients got their podcast episode, then it works. But we have no way of knowing. So if our listener here didn't get Monday's news episode for some reason, let us know.

25:57Because. Oh, no, they did. Well, they might have. I mean, the URL is correct, but they are going to the new application instance, which we're not tracking. Which goes to the new CDN. Which has the new CDN, which has the pipe stream. Yeah. Same data. Just different applications. The same data. The data will be the same. Okay, let me tell you the other thing I've noticed. Okay, go on. So that's one. Let's debug, live debugging. Love this. Two you already know about, which is that, and this is probably the exact same issue, is that when we posted our auto posts to, I think Slack in this case, posted the app instance URL, not the change.com URL.

26:33I thought it was Zulup. Wasn't it Zulup? It might have been both. Actually, it was both. Yeah, it was both. And so there was a URL mismatch, which I think is the exact same issue. Yep. And then the third one is that I subscribe to all of our feeds because I want to make sure they all work. And so whenever we ship an episode, I get like five versions, you know, just patent our stats, getting five downloads for the price I want. And specifically the slash interviews. So yesterday's show with Nathan Sobo. Two days back as far as we ship this, but yesterday when we record, it went out and I downloaded on the changelog feed and I downloaded on the plus plus feed.

27:13and I didn't download it on my interviews only feed because you can just get the interviews if you want. And that feed did not have that episode until this morning when I logged in and said, refresh the feed. And I forced it to refresh that feed and then I got it. And so there's, and again, that's probably, those are background jobs. So somehow that did not get refreshed. So that's the third thing. Okay. The fourth one. Okay. There's four. Yesterday I disabled Slack notifications entirely and this is our last step to cut entirely over to zulip and i have a blog post which is going out announcing that we're no longer on slack don't don't go there however after adam shipped that episode it posted the new notification into slack even though i that code doesn't exist anymore and i deployed it so i'm guessing it still exists on your yes your experimental one is not keeping up with code changes yeah okay so all my bugs are related to this very exciting deployment that i didn't know we broke it i think we broke it i think so yeah yeah i think we broke it so those are the four things i've noticed no sorry i broke it let me take responsibility for this yeah that's much more fair i had nothing to do with it well friends i'm here with terrence lee talking about what's coming for the next generation of roku they're calling this next gen FUR.

28:36Terrence, one of the biggest moves for FUR in this next generation of Heroku, it's being built on open standards and cloud native. What can you share about this journey? If you look at the last half a decade or so, like there's been a lot that's changed in the industry. A lot of the 12 factorisms that have been popularized and are well accepted, even outside the Ruby community, are things that are, think, table stakes for building modern applications, right? And so being able to take all those things from kind of 10, 14 years ago, being able to revisit and be like, okay, we help popularize a lot of these things.

29:09We now don't need to be our own island of this stuff. And it's just better to be part of the broader ecosystem. Like you said, since Heroku's existence, there's been people who've been trying to rebuild Heroku. I feel like there's a good Kelsey quote where we can stop trying to rebuild Heroku. It's like people keep trying to build their own version of Heroku internally at their own company, let alone the public offerings out there. I mean, I feel like Heroku has been the gold standard. Yeah, I mean, I think it's the gold standard because there's a thing that like Heroku's hit this like piece of magic around developer experience, but give me enough flexibility and power to do what you need to do.

29:44OK, so part of fur and this next generation of Heroku is adding support for.NET. What can you share about that? Why.NET and why now? I think if you look at.NET over the last decade, it's changed a lot. .NET is known for being this Windows-only platform. You have WinForms, use it to build Windows stuff, double IS. And it's moved well beyond that over the last decade. You can build.NET on Linux, on Mac. There's this whole cross-platform open source ecosystem. And it's become this juggernaut of an ecosystem around it. And we've gotten this ask to support.NET for a long time. And it isn't a new ask.

30:22And regardless of our support of it, like people have been running.NET on Heroku in production today. There's been a mono bill pack since the early days when you couldn't run.NET on Linux. And now with.NET Core, the fact that it's cross-platform, there's.NET Core bill pack that people are using to run their apps on Heroku. The kind of shift now is to take it from that to a first-class citizen. And so what that means for Heroku is we have this languages team. We're now staffing someone to basically live, breathe, and eat being a.NET person, right? someone from the community that we've plucked to be this person to provide that day zero support for the language and runtimes that you expect and like we have for all of our languages right to answer your support and and deal with all those things when you open support tickets on heroku and kind of all the documentation that you expect for having quality language support in the platform in addition to that one of the things that it means to be first class is that when we are building out new features and things is now one of the languages as part of this ecosystem that we're going to test and make sure runs smoothly, right?

31:23So you can get this kind of N10 experience. You can go to DevCenter. There's a.NET icon to find all the.NET documentation. Take your app, create a new Heroku app, run Git Push Heroku Main, and you're off to the races. So with the coming release of Fur and this next generation of Heroku, .NET is officially a first-class language on the platform, dedicated support, dedicated documentation, all the things. If you haven't yet, go to heroku.com slash changelog podcast and get excited about what's to come for Heroku. Once again, heroku.com slash changelog podcast.

32:06Okay, so let's talk through this in terms of what a potential fix would look like. We have a new application instance which behaves as production from all purposes, right? Like the content is exactly as production. It connects to the same database instance. It has all the same data. What isn't happening is the code updates aren't going out automatically. That has not been wired because my assumption was I will only deploy this one instance. I'm going to change a couple of properties so it has the new CDN configured and I'll see how it behaves the whole stack in isolation. what what happened obviously is the new instance is consuming the same jobs the same background jobs as the existing production so very helpfully it has sent the new links which are all temporary especially like the application links the ones that you've seen in zulupe and a couple of other places which are just for the application origin and they are only meant to be there for the cdn everything should go through the cdn but the cdn hasn't been configured yet through everything because that's like where the test comes in how does the application behave so some links need to be application links how does the cdn behave so on and so forth so in this case we need to somehow fix those links the ones that went out and they're incorrect i'm not sure whether we know what they are and if not then we need to basically make the new this experimental application instance not send not consume basically jobs not process any background jobs we just need to disable oban in that one perfect and then it would never get invoked unless you manually go to the website right yeah and then we want to make sure that nothing crawls it yes that's then they'll start sending traffic to its endpoints instead of our main website so let's do that tootsuite so i think yeah i think i think we're finished with the recording let's go and do it no no we haven't don't worry this is this is still going okay so yeah but that those two changes i think will mitigate the current issues yes yeah that sounds about right okay so that makes me happy as long as we get those rolled out here we figured it out we figured what the issues are all right so what do i want to do now i think i would like to see how many pipely instances we're running all over the world okay and for this i'm going to use a new terminal utility which i found that i i like i was like yes this is exactly what i was missing it's called fly radar fly radar okay this is what it looks like i'm going to go to that it's all ncurses based it's all happening in my terminal and it's beautiful cool fly radar 021 i can see all the changelog applications the one that we're going to look at is a cdn so by the way the two applications do you see this one the changelog 2225 0505 is the new application instance that was deployed three days ago while the one above the 2024 that is the current production and that was updated one hour ago so the code will differ and the slack notifications if this application instance picks up a job it will do whatever it's configured to do okay it will be the wrong thing another thing we could do briefly before we figure that out is we could just redeploy that one so at least it's current yes and it won't do any slack notifications because i definitely don't want to say we're no longer doing slack notifications and i have another one come in and i'll have egg on my face as soon as we stop recording i'll go and do that not a problem okay so let's have a look at the cdn 2025 0 to 25 which is the instance when it was deployed and it has had a few updates what do we see we see 10 instances you see the region and you see it's been updated one day ago i see sydney is that right yes i see chicago yes lhr is that the virginia one heathrow oh london heathrow of course yes these are the airports by the way uh jnb that's uh joeberg johannesburg johannesburg very good uh san jose yes correct okay iad that one's that's the virginia one that's the one okay that's the one adam adam you want to guess these sin i know dfw what is dfw oh that's where you're down to fort worth and i think france that's fr a is probably france is my france what's that's actually frankfurt oh frankfurt germany sounds too singapore singapore singapore cool oh man i keep seeing this okay and scl i haven't i don't know what scl is come on i keep sitting that are okay i don't know all right let's do fly ctl uh platform i think uh regions i think and the regions list there we go san diego chile yeah that's the one scl uh yeah that's the one santiago that was it scl that's how we see what the regions are cool that's cool man yeah get over there in australia or new zealand or something well we we well we do have sydney so that's true we can add yeah we can add more i mean we had 10 but uh we can add no no no sydney covered that i just forgot about sydney yeah so you've seen all the machines and in terms of other uh uses like it has logs alpha logs so this is something that's really cool so these are the logs for the new let's see what logs we have what requests we have flowing to the new changelog instance this is a cool tui congrats to the fly fly radar coder author person this is cool it reminds me of canines yeah exactly that's exactly it yeah that's exactly it oh look we have some requests robots robots got some requests and the home page got some requests and this is iad iad so we can see what instances were requested okay so now let's go to let's go to the um i'm not liking these requests gerhard how can we get requests yeah exactly well we will be getting because we have the cdn we have um monitors set up we have a bunch of things now these are the requests going to the existing you can see there's a lot more traffic going to the to the existing application yeah if you ask me there's too much traffic the cdn is not in his job that's what we're trying to fix right right there's way too many requests hitting it and you can see that the regions right this is uh we have two regions ewr adam what does ewr stand for do you know oh why uh right on right on okay yeah perfect that that's exactly what it is that's right on man yeah so so so we can focus only on like like specific instances to see the log so i i think this is really cool so we've seen this let's move on furkan kli fly radar furkan kalasio glue that's quite the day furkan kli yeah so he he built this that's um i think it's a really cool tool we can go and check it out on github it's all written in rust so it's really really fast it's a terminal ui it was inspired by canines yeah yeah that's it so um issue five uh march 22nd that's when i um i i just stumbled across it so i captured it you can go and check it out um but it was really cool like when i when i seen fly radar i thought like wow this is exactly what i what i wanted anyway anyway back to the pipe dream so which backends do you think serves the most requested url another question we have three backends or three origins right you have the application origin the one that we've been focusing on there's a feeds back end and the assets back end so in the last seven days which backends serve the most requested url like the one top url if you also don't know what it is which one serves that particular that's the question yes okay i don't know there's only three possible answers yeah i'm gonna go with feeds same feeds feeds so apparently we're serving this podcast original image about 10 000 times per day or once every 10 seconds that's the assets endpoint i had to check what it was yeah it is assets yeah it's our change log that change log so the answer was was assets actually i guess that makes some sense because everyone has to download that into their podcast app all the time yeah cache that sucker come on i know right do better things yeah do a better job with caching it that that would that would be a good thing so but honestly um it was the second one i guess feed so we were almost correct if it wasn't for that one image i'm wondering how does the new cdn behave for our most requested url which is not a static asset so how does it behave for podcast feed i'm going to run three commands actually a few more than three the recording has been done so if anything doesn't work as it should we'll switch back to the recording but that's going to be a backup all right so let's go back into the terminal and we'll experience this firsthand just to see what it feels like so i'm in i'm in the pipely repository and the first command which i'm going to run is just debug and by the way anyone should be able to clone the repository and do exactly what i do what's happening here is behind the scenes it is building everything that we need for the cdn including the debug tooling and it will run it locally and the tui that you see here because it is a tui it has a couple of shortcuts is dagger so all this is wrapped into dagger okay so i have a terminal opened in pipely all running locally all right um so the first thing which i'm going to do is i'm going to benchmark the current cdn changelog.com so i'll do just bench cdn all this is wired together is sending a thousand requests to the feed endpoint and this is what we see so the current cdn serves about 300 requests per second and it's the size that is the interesting one the size is about 220 maybe bytes per second so i think that the cdn is faster but the bottleneck here is my two gigabit home connection and this is as much as i as i can benchmark it so that's the limit so if we were to benchmark using the same connection cdn2 this will go to the pipe dream to feed this is how that behaves and by the way this is live real traffic that's happening here so 177 and 132 megabytes per second so what do you think is happening here if you had to guess well my guess would be that it's not as much bandwidth as fastly has that is correct yes so i'm looking at fly here right and this is the cdn instance we have the different instances do you see here like london heathrow that is the one that lit up lit up in response to me sending it a lot of traffic and you can even see it here right if i do london heathrow you can see that's the one that was serving the most bandwidth.

43:33And actually what I've hit is the 1.25 gigabit limit of this one instance. And that's just a constraint of the actual instance on Fly, like that particular Fly VM or whatever they're called. That is correct. Yeah, exactly. So if I do machines list, FlyCTL machines list, you'll see that, and let me just do an RG on LHR, you'll see that we have like a single instance in Heathrow. We could run more, and that's what we're going to do here to see if running more instances will increase the bandwidth. So I'm going to do, let's do flyctl scale count three. And I'm saying, we're just basically going to run three instances in the Heathrow region.

44:13The reason why we don't do this is we'll just add cost. When we are in production, we may need to do this because some areas may be running hotter than others. So we may need to scale it accordingly. But right now, every single region has one instance only. so uh let me do machines list so what i want to see is they are all started and they're all running the health check there is one yep these are all good um yeah everything is nice and healthy so now let's go back and let's run the same benchmark and you'll see it live okay so still the same thousand requests to the feed endpoint and 180 so just about the same not much has changed it takes a while right for everything to warm up and the request to be spread correctly we've seen there a blip so let's see how does it behave now okay so we're 150 megabytes per second if we run this a few more times so that everything is nice and spread that was request per second right you said megabytes per second that's request per second so this is so 171 megabytes per second which is almost like 1.7 gigabits and the request we have 228 so these three instances that's what we see and if we run this enough times when i tested this uh last time i was able to get to uh that's about two gigabits but it's not um it's not like an exact result every single time based on network conditions based on a bunch of things you know based on where those instances are placed within the fly network but three instances and even when i added more i've seen there was like this limit like obviously like the the two gigabits well you max out eventually right exactly i max out eventually i'm still not maxed out currently and the reason why i know that is because i bench see if i bench cdn2 i can see that that brings me the close to that two gigabits right 220 cdn1 this is fast cdn1 yeah this is changel.com this is fastly that's correct and those 300 and something requests per second so fastly is still faster because we haven't added enough instances in your region in order to get our bandwidth up on fly to max out your gearhard personal bandwidth exactly exactly okay so adding instances doesn't really move the needle very much but it does move it eventually if you really wanted to exactly so this is this is maybe even the question to the fly team so when it comes to the instances if i look at what instances we provisioned you can see that we are running shared cpu 2x and they get two two gigabytes of ram the question is and i think we kind of like touched upon this last time even the performance instances we don't seem to be getting more bandwidth there is a point at which an instance doesn't get more traffic and depending on maybe the region's capacity maybe there is some sort of limit that we're hitting now do you remember bunny yeah yeah okay we can we can bunch bunny which is still live or bunch bunny bench bunny we can bench bunny and bunny will go and this is how that behaves bunny changelog.com bunny doesn't let you right exactly so the rate limits me so i can't benchmark bunny you think that's because they don't want to be benchmarked or you think it's because they're just fighting off spanners.

47:33I think it's throttling, yeah. They are throttling. So bonniechangelog.com. And I have been benchmarking them quite a bit in preparation for this. My IP might be blacklisted somewhere on the body side. Yeah, but that's the reality. Cool. You should be able to get some sort of pass. Like, hey, I'm a developer and I'm testing things. Right. Yeah, just benchmarking. Of course. Yeah, I think so. I think so. Cool. Okay, so I'm wondering, if i had a hundred gigabit internet connection and one day and this is a fact one day i will have that internet connection and fly did too right because remember fly i mean in this case flies is the bottleneck correct what could we expect from pipedream just up runs the whole of pipedream locally okay so now you got no band yeah no network no network exactly it's just like the It's everything is running on the same host.

48:32And you can see that this is actually forwarding traffic to the feeds endpoint, to the static endpoint, to even the application origin. This is like all of our features. So it's all here, right? It's all here. It's all here. So let's do bench feed and let's see what we get. Oh, we're getting massive amounts of... That's 200 ,000 requests. That is 200 ,000 requests. Yes, it's more. What do you see in data? Can you read that out for us? 85 gigabytes. Right. That was a bit silly, but yes, it's every 10 seconds. So now it's switched, because we had so many requests, the scale switched from one second to every 10 seconds.

49:09And this is what we see. We are pushing 11 ,000 requests per second, and we're transferring eight gigabytes, not gigabits, gigabytes per second. So if we have a really fast network, we could saturate close to 100 gigabit. That's insane. Yeah. So the software works. And that's just a credit to Varnish, right? Pretty much. Yeah. Yeah. It really, really works. It really works. When you hold it right. When you hold it right. And you don't have a network. Sorry. Exactly. Well, you have a hundred, you need to have a hundred gigabit connection. So that's, I think that's the hard part. And Fly needs to, Fly needs to have, or whatever provider we run, it needs to have more network capacity.

49:55Because right now my internet is faster than what the Fly instance does. Yeah. And I can't saturate it. And we've seen because I can saturate, I can saturate fastly. Cool. So, and I think the interesting thing, which, which I haven't shown yet, and I can, I can, I could because it's behind me, but anyway, that that's not very visible. What I would like to show is basically I'm hitting the limit of my CPU, right? Like where I'm running this benchmark, it's a 16 core machine and I'm running both Varnish and the benchmarking client, oha oha in this case and between the two of them they're saturating 16 cores and that's what we see here so the bottleneck really is the cpu it could go faster because again networking is just all in the kernel so pipe dream and pipely is an iceberg and we explored just the tip of it so most of it is underwater are you talking about lines of code no i'm talking about many things but let's go so i'm wondering how how many of my uh 20 lines of ballooned into at this point it's there it's there that thing is coming up so yeah stay tuned stay tuned so vtc stands for uh varnish test case okay and pontus algren algren oh yeah i saw this comment yeah so pontus algren one of our kaizen listeners mentioned this in azula message back in december 2024 so he said regarding the testing of vcl did you consider the built-in test tool vtc so you were doing something else previously i can't remember what you were doing we are still doing that but i'm also doing this okay so uh i'm just going to play the recording okay okay it's it's just easier so just test vtc is going to run in three seconds all the tests for the different varnish configuration that we have for the live stream cool this is really really fast this is the equivalent to your unit tests if you wish you're running the test against like production instances last time i was and now you have to do that are you still are still there yes why why wouldn't you replace it hang on let's just give it a minute we're getting there so this is so this is what the v what the vtc looks like and basically you can you can control it at a very low level in terms of the requests the responses the little branching so think of it when you're trying to come up with a final varnish right you make like little experiments to see how the different pieces of configuration would work and that's what vtc enables you to do you can write a subset of your vcl you can configure clients you can configure servers and you can make them do things in an isolated way in a very quick way you can basically model what the thing is going to look like and you're going to check if what you thought would happen does happen and that's what makes it really really fast and it's all built into the language so it's it's there and we have it and it gives me a nice tool to figure out what is the minimal set of varnish that i have to write for this and i think this is where like that number of lines of code and number of lines of config comes in.

53:11But we all know that we want acceptance tests. We want to see what users will experience. And remember, this is what you were asking for, Jared. You were saying, how do we know that this new thing is going to behave exactly the same way as the existing thing behaves? So what we now have is you see the test acceptance these are all the various things that we can run in the context of pipely we can do test acceptance cdn test acceptance cdn2 or test acceptance local and this is using uh hurl and we're describing the different scenarios that you want to test for real testing these real endpoints which one would you like us to try out local local great so what i've heard is changelog that's exactly what i said change log okay okay so whatever it is i don't know why you even asked well you have to have a bit of fun so test acceptance cdn and test acceptance cdn is going to run the same tests against the cdn it's going to test the correctness of our cdn not using vtc though say again not using the vtc stuff no this is hurl this is hurl stuff this is like tests exactly this is like a different level the vtc stuff is just for the varnish config hurl in this case the acceptance tests are doing real requests and checking the behavior of the real endpoints like for example am i getting the correct headers back am i being redirected is this returning within a certain amount of time what happens if i do this request twice how does it behave is it a miss versus a hit what happens so we have 30 requests that we fire against the existing cdn and we see how it behaves and then what we're going to do we're going to run the same requests against the new cdn and it's slow why do you think it's slow Well, I don't know what these tests are doing.

55:19So I can't answer that question. So these tests are checking the behavior of the various endpoints. For example, the feed endpoint or the admin endpoint or the static assets endpoint. In this case, you can see that we are waiting for the feed endpoint. So if you go back and you think about the various delay and the stale versus miss, we are checking how the stale behavior um sorry we were we're checking how the stale properties of a feed responses behave so if i'm going to hit this endpoint within 60 seconds will it show up as stale Will You so we're literally we're checking and we have to wait to see will it expire will it refresh so so you're delaying on purpose to see exactly i'm delaying it on purpose and it takes about 70 seconds because we need to wait that long right to to to test the staleness and by the way that's that's something which i'm going to do next so we're going to check the staleness of something and the staleness currently set to 60 seconds and you can see we can like we can do the variable delay so this is the real cdn we're going to pipetream we're not testing the local one we're testing the pipe dream one and this is the existing configuration which we consider to be production now you said local and now we can do the same tests we're going to run them against local and we're going to change a couple of properties because locally we want slightly different behavior and what we care about is that speed right we want these tests to be much much quicker and in this case you can see like the actual requests going through you can see the responses you can see the headers we still are testing delays but the delays are much shorter which means that the test will complete much much quicker so we we control these variables and production is just like you know as as it is this is how it behaves and that's what we're testing so it'll be slightly slower shall we do it for real would you like me to try to run another test and see how it behaves if i do the acceptance local or shall we move on to something else what is the conclusion from that like conclude some things for me well the conclusion is that we are able to run the cdn locally and poke it and prod it and make sure that the cdn in this case is behaving exactly as we expect it to we have a controlled way of configuring everything what i mean by that i mean the backends the various backends that we use we have properties to control like ttl stainless freshness and see how different configurations change the behavior of the system we also have it deployed and we can check if the existing cdn behaves the same as the new cdn i haven't written all the tests only like the big ones does the feed endpoint behave correctly do the static assets behave correctly what about the admin endpoints or those that shouldn't be cached do they behave correctly so i'm starting to build a set of of endpoints and set of tests that check how those endpoints behave and there's certain differences right like one cdn behaves slightly differently we know like the existing one right that we're trying to improve on so we can see where does it fall short there's a couple of interesting things that we can look at for example um i've seen that we for example don't cache the json variant of the feed of the rss maybe would want to do that i don't know but going through this like testing the correctness of the system made me look into parts where i wouldn't normally look the best part is that we can run this locally we are in full control of everything that happens in our cdn it's a lot of responsibility and it takes a certain level of understanding to know what the tools are and how they fit together but we have it yeah that's awesome because now we don't have to just poke at a vcl in the sky and hope that it does that's right and just only test in production that's it yeah you can actually make changes with confidence is that a state of the art for any of the cdns out there like can you do this level of acceptance test between i guess you probably can't right we can't run fassy locally we can't run even bunny locally we can only run our own thing locally so you can't really test the way you'd develop it locally and then develop it in production but you can test you know x y cdn versus pipely or pipe dream right you can test that that's what you're doing right now i think the first step is to being able to run it locally and running anything of that magnitude locally is hard let me rephrase that i would say if you are unhappy with your cdm provider thus far has there been a way to say what the original question was can we trust moving to something else in this case that something else is something we've built not a different public provider and so we're scrutinizing a little bit more but if you were unhappy with you know one cdn and you were thinking man i want to move to a different one has there been a state of the art to test the i guess the efficacy between different cdns has this tooling been there before i'm not aware if it has if someone from our listeners is aware of such tooling existing i'd love to learn about that i think it pretty much comes down to diy as in how much of the correctness of the system are you testing for and in this case even though it is a cdn um it is part of our system right because it determines how the change log website and the application and all the all the origins behave ultimately how do users perceive them and the best thing that we have honestly are the logs because based on the logs you can see what users experience but is that good enough i mean these systems are really big right like global scale big it's really hard for example even for me i mean sure i could force and test every single endpoint but on like when i'm running these these tests right when i'm for example testing changelog.com i'm testing whatever wherever i'm closest to based on the network conditions based on whatever's happening and i need to encode certain certain properties i care about to check that they are behaving correctly the same tooling could be used for any other cdn so once we encode the things that we care about in terms of the correctness of the system let's say that one day we migrate to cloudflare if we did that we would run the same set of acceptance tests against cloudflare or whatever we're building there and see does this thing behave the same as the thing that we're migrating from so there are like these harnesses that we are required to have to make sure that that the systems behave correctly because they're big, complicated systems.

1:02:21And most of them are beyond our control, as we've learned over the years. Does that answer your question, Adam? Kind of. I mean, I think it does. I think what I was pointing at or potentially trying to uncover is the potential of, you know, we're all allergic to vendor locking, essentially. You know, I feel like I wonder if there's a level of vendor locking because you don't know unless you make the move. And it's hard as a developer and I see or even a VP to say, we got to make this change. We've got to move to a different platform because of X, Y, and Z and whatever their data is, whatever their reasons are.

1:02:54And I wonder how many people or how many teams are staying where they're at because they have fear of the unknown. The unknown is that they can't test to this degree, this acceptance level. I mean, yeah, that is real. I mean, just think about the journey that we had to take to get to the point where we are today. it took a lot of effort it took a lot of time it took a lot of understanding what even are the components and we could have picked something else we didn't have to pick varnish but we didn't want at least i didn't want to change too much at once one day we may replace varnish it is possible the real value is in understanding what the pieces are and how they fit together whatever those pieces are whether it's kubernetes whether it's a pass whether it doesn't really matter it's a database take your pick each context is different so then how do you go about understanding what the pieces are how do they interact and how do you ensure i think this is coming back to where we started how do you ensure that what we do does genuinely improve things and that is the hard part being able to measure correctly being able to understand what improvement even means in the first place is really hard and what trade-offs are you okay to make we take a lot of responsibility by running this ourselves and i'm very aware of that i think that is that is really like the hard part being confident that you can pull this off having the experience that you can pull it off and you can learn anything that you're missing and if you apply those principles to whichever context you operate in you'll be good it won't be easy but you'll have learned so much

1:04:39Well, friends, I'm here with a good friend of mine, David Hsu, the founder and CEO of Retool. So, David, I know so many developers who use Retool to solve problems, but I'm curious. Help me to understand the specific user, the particular developer who is just loving Retool. Who's your ideal user? Yeah, so for us, the ideal user of Retool is someone whose goal, first and foremost, is to either deliver value to the business or to be effective. Where we candidly have a little bit less success is with people that are extremely opinionated about their tools. If, for example, you're like, hey, I need to go use WebAssembly, and if I'm not using WebAssembly, I'm quitting my job, you're probably not the best Retool user, honestly.

1:05:25However, if you're like, hey, I see problems in the business and I want to have an impact and I want to solve those problems, Retool is right up your alley. And the reason for that is Retool allows you to have an impact so quickly. You could go from an idea, you go from a meeting like, hey, you know, this is an app that we need to literally having the app built in 30 minutes, which is super, super impactful on the business. So I think that's the kind of partnership or that's the kind of impact that we'd like to see with our customers. You know, from my perspective, my thought is that, well, Retool is well known.

1:05:55Retool is somewhat even saturated. I know a lot of people who know Retool, but you've said this before. What makes you think that Retool is not that well known? Retool today is really quite well known amongst a certain crowd. Like I think if you had to pull like engineers in San Francisco or engineers in Silicon Valley, even, I think it'd probably get like a 50, 60, 70 % recognition of Retool. I think where you're less likely to have heard of Retool is if you're a random developer at a random company in a random location, like the Midwest, for example, or like a developer in Argentina, for example, you're probably less likely.

1:06:30And the reason is I think we have a lot of really strong word of mouth from a lot of Silicon Valley companies like the Brex's, Coinbase's, DoorDash's, Stripes, et cetera, of the world. There's a lot of chat. Airbnb is another customer. NVIDIA is another customer. So there's a lot of chatter about Retool in the Valley. But I think outside of the Valley, I think we're not as well known. And that's one goal of ours to go change that. Well, friends, now you know what Retool is. You know who they are. You're aware that Retool exists. And if you're trying to solve problems for your company, you're in a meeting, as David mentioned, and someone mentions something where a problem exists, and you can easily go and solve that problem in 30 minutes, an hour, or some margin of time that is basically a nominal amount of time.