Kaizen! Pipely goes BAM (Friends)

44:28It's just the exploration of the hardware. How does it work? You know, what works together? yeah me too me too and i think um we're at a stage where it does make sense to have a few lying around have a windows machine right if you have to do testing or anything like that use the linux machine you know i think it's a very eye-opening experience as to what is possible and then you know if mac is your default or if not if you have the opportunity to get maybe a mac mini do that as well and then you will find the one that you love and the one that's your daily driver and you have a couple as backups you know when something goes wrong because it does it does happen cool yeah man well talking about podcasts because in video podcasts because that's how we started uh i don't think we finished there's like so many new features around youtube and around content on youtube um i think the reactions have been mostly positive there was a whole zoolib discuss about it which i don't contribute to many but this one i did contribute to and uh february 1st i even got some um love hearts from a few of you nabeel and marsh so thank you very much for that but uh what do you think about launching video podcasts like how was that transition how's that new chapter i think it's going pretty well i guess i don't consider it to be over with maybe it is because i guess a lot of what we think about is production workflow and we're constantly trying to improve that and make it better.

45:56I would say that we successfully went video first now and we have systems in place that we can do that reliably. I had to build a few things and we had to figure out a lot with regards to chapters and timestamps and how we handle the videos on YouTube versus the podcast, episodes, and audio. And all the nuts and bolts, I think, were fine. We just kind of figured it all out and did it. Nothing really was too difficult there. The response has been positive. I think a lot of our audio listeners have a little trepidation because they think, is it going to become a YouTube show? And, you know, they never want to listen to it on YouTube, which I don't either, honestly.

46:41We're doing this for people who like that kind of thing, like Gerhard, I guess, and others. And I acknowledge that you all are out there and we appreciate that you are. And we want you to watch it on YouTube, which is why we came there. But our existing audience, very few of them find much value in the videos, I think. Or the ones who at least are vocal don't. And I get that. And of course, the trepidation is like, well, will the audio suffer? And will we start to pull a thing up on the screen and have reactions to it without explaining what we're looking at? I don't ever want to get there. Hopefully we can be self-aware and always remember that we have a listener, not just a viewer.

47:19and explain what we're looking at if we are looking at something. So for them, I understand because if you love something and it's changing, you just hope that it doesn't change for you for the worse. And so hopefully we haven't done that. I think we've had most people who had trepidation, at least so far, have been fine with the change. They haven't noticed much of a difference. For those who love video podcasts or watching conversations on YouTube, because is it a podcast actually? i guess youtube thinks it is um we're there now and people are watching you know we get 500 to a thousand watches on a on a video we hope to grow that and um no real complaints there i don't think besides your random youtube troll which we've had trolls our entire career so we don't feed them or care about them very much that's my initial thoughts adam anything to add I ran into because I was actually talking to my son last night.

48:20I was like, dude, because my son's nine, and I'm about to give him an Ubuntu desktop machine to play with. I'm going to start teaching him Linux. And I was, like, excited because I had just SSH, like, literally maybe earlier that day, SSH into this Windows machine. I was like, success, you know? And I was, like, referencing embedded systems and why it's so cool, how Linux is so cool. And I'm like, you want to see something cool? And I went to YouTube and I searched embedded changelog. And I just searched those two things and it came up with the embedded podcast we did, Jerry, that you're aware of.

48:54And I go there and there's this comment that's like 500 words deep. And I'm like, I had no idea, one, that this comment was here. And two, that is like, you know, I was re-revelationed, I suppose, in terms of like how cool this move is, is that we've got this new commentary level and the person's like, I like this podcast. I'd love to hear more. And they kind of go into all this stuff. Now, the person doesn't have a username. They don't have an avatar. So that's kind of sad. But, you know, I'm still hopeful that there's more like that that are thicker. Jeez, y 'all don't like that word. I don't just like it.

49:34I think thick is a good thing. Anyways, I won't go back there. It's an exhaustive, you know, thoughtful comment that I haven't even read the whole thing yet. But I was like, wow, there's this super huge comment that somebody's like actually talking about relevant things and not how we suck. So that was cool. I love that. I like the, you know, I was pushing for this because I was like, this is something, this is what we need to do. There's a whole audience there that we can tap into that we're not. And clips are great, but they're not the full length podcast. I'm now sad that there's that when I share with people we're on YouTube that they're like hey did you just start producing this podcast I'm like nah man it's been like forever basically and so we have this huge backlog that's not there and that kind of makes me sad because there there's a lot of visuals and a lot of just like seeing the reactions like Garrett mentioned with Tim just being able to see his pause or his thinking or my thinking whenever I'm talking or him pointing to his mini stacks behind him, I think that's, it's not for everybody, but I think there's a large majority of people who are gravitating more and more towards that, who do listen on YouTube, pay attention when they want to, but when they want to, they can go and look at the screen.

50:57You know, and that's been my use case for it personally. And so I wanted that for us for so long, and I just felt not so much bored, but there was a missing necessary humanistic component that was visual that wasn't there and so when you're audio only I feel like you're stuck in this this box and I feel like we're now like we're like the genie out you know we're the cats out of the box so to speak we're we're able to explore the bigger world of YouTube and capture not so much more of an audience but like I think there's a lot of people there waiting wanting what we produce and now we're there in full form yeah so youtube here to stay um a new way to interact for sure and more and more integrations in in the website i quite like that for example like the watch button that was something which was one of the new things to drop yeah on an episode uh it's getting a bit crowded uh maybe this one and but it's there you can click on it is you can click on that and it'll pop in there and just look at that how amazing is that cool yeah it's cool right that's the I like that right there.

52:05And on the play bar, if you go to an episodes page, the play bar got a little wider and it has a watch button, which will do the same thing. It'll pop, it'll embed it underneath it. Once you click on it, we don't auto embed because, you know, only when you want it on demand. Should they say listen? This is watch. Yeah, maybe. Listen and watch, maybe, yeah. Play and watch, maybe listen and watch. Yeah, that'd be a good improvement. Yeah, but these are nice. Like you can watch it right here and they just get automatically expanded. I like that we are a commit-driven company, by the way. A lot of the features that get dropped, I just find them through commits.

52:43This is so basic. No problem and circumstance, you know, no blog posts, nothing. No, just we are a commit-driven company. So if you want to know what is happening at Changelog, follow the repository and just like look at the commits. That's right. We're very committed.

52:58Well, friends, I'm here with Samar Abbas, co-founder and CEO of Temporal. Temporal is the platform developers use to build invincible applications. But what exactly is Temporal? Samar, how do you describe what Temporal does? I would say to explain Temporal is one of the hardest challenges of my life. It's a developer platform and it's a paradigm shift. I've been doing this technology for almost like 15 years. The way I typically describe it, imagine like all of us when we were writing documents in the 90s, I used to use Microsoft Word. I love the entire experience and everything. But still, the thing that I hated the most is how many documents or how many edits I have lost because I forgot to save.

53:40Or like something bad happened and I lost my document. You get in the habit when you are writing up a document back in the 90s to do Ctrl-S, literally every sentence you write. But in the 2000s, Google Doc doesn't even have a save button. So I believe software developers are still living in the 90s era, where majority of the code they are lighting is they have some state which needs to live beyond multiple request response. Majority of the development is load that state, apply an event, and then take some actions and store it back. 80 % of the software development is this constant load and save.

54:14So that's exactly what Temporal does. but it gives you a platform where you write a function and during the execution of a function of failure happens, we will resurrect that function on a different host and continue executing where you left off without you as a developer writing a single line of code for it. Okay, if you're ready to leave the 90s and build like it's 2025 and you're ready to learn why companies like Netflix, DoorDash, and Stripe trust Temporal as their secure, scalable way to build invincible applications, go to temporal.io once again temporal.io you can try their cloud for free or get started with open source it all starts at temporal.io now that we've got video first going on it's time to get cpu officially launched turn that frown upside down into a smile and an index you know something that's cool yeah very nice okay any infrastructure that we need to think about talk about for cpfm you could share with him jared what your thoughts are on the application the plan is just to have a stand a bog standard web app with a with rss feeds right nightly style like that is a bog standard app or oh in terms of the actual software do you have a database do you need a cdn what would that There will be a database.

55:36CDN will probably be smart, but maybe we just drop it on R2. Probably similar to what we're running now for us, only it's going to be simpler and it's going to be a separate software stack. So probably going to go back and give Ruby on Rails another kick down the road and see. Interesting. Just because it's been a long time and I've been in Elixir land for almost 10 years now. And every time I write a little bit of Ruby code, I'm like, you know what? This is my first love. And so probably going to be a Rails app. deploy it on fly it'll be pretty simple have a back end you know write out html pages and rss feeds that's that's the plan so far i haven't written a lick of code yet so okay these things may change but that's the plan nice keep it simple okay yeah um neon for the database i'm imagining yeah i would probably just reuse all the stuff that we've been using over here okay yeah um public repo, private repo.

56:35good question good question probably public i don't see why not yeah okay i'm not gonna promise that but I can't think of a reason why it wouldn't be public. It's mostly the admin's going to be for like just managing the podcasts that are part of it. And then the code, the actual logic of it is going to just be in building basically a super feed for people. And maybe custom feeds too. So you can get your CPU pods that you like. And maybe if you don't like one, uncheck it or something. I built that already for us. So rebuilding it over there would be straightforward. which will require user accounts, of course.

57:14Or would it? Maybe not. I don't know. I'll figure that out. That's the plan. Pretty straightforward. Not much code. I don't see why we wouldn't open source it. Unless I'm really bad at Ruby now. It's been a long time. It's embarrassing. I think that's more of a reason to open source it. You can ask for help. Contributions welcome. I haven't typed Rails new since probably 2015. So I'm kind of excited to like just type Rails new and see what happens. Well, make sure you record that. I think many people will be interested in your reaction. I will. Gerhard's thinking, he's thinking about the content.

57:47That's what he's trying to ask you about, Jared. I know. He's like, how can we promote all these cool things? Yeah, that's what I'm thinking. Yeah, I mean, maybe record it. I don't know. I guess if you want that kind of content from me as I build out this new web app, which honestly is not a super exciting web app, but it's still maybe i'll use cursor the whole way you know and then i'll just uh curse my way along and then just rewrite it myself if you want that let us know in zulip let us know in zulip yeah it's a new world and i think seeing how how you would approach that with your rails knowledge things that have genuinely improved from how you remember it what is better what is worse because you have a unique perspective which is the elixir one running an elixir application for so many years how does that compare to ruby and rails i don't think many people did a switch back i hear i keep hearing about people going from ruby and rails to elixir but going back i'm not aware of anyone doing that it didn't make the hacker news it didn't appear on the changelog this would be news well i wouldn't be ditching elixir because we'd still keep changel.com over there so i would be going back for a new app but you know living in both worlds from that on which i'm happy to do um yeah that could be interesting you know old man fumbles around in the dark with rails you know yells at rails and then he'll some more okay yeah that that sounds interesting cool cool so pipely let's see how long this is going to take and by the way this is like where where where the where the screen sharing will um get into its own so there was a question that we had uh from team tim ucken i'm not sure if i'm pronouncing that right u c k u n why do you need a cdn if you have fly.io and i replied to nzulip that's like the sort of conversation that happened there and i went through all the various things so the reasons why we need a cdn even though we have fly so you can read it either in this github discussion or in zulip it's all there so you can go and check it out but the thing which i would like to talk about is that we are starting to have contributions to pipely and you may be wondering pipely what pipely well what is pipely we rede we renamed from pipedream to pipely why because pipedream is taken we can't get pipedream.com we've already established that that's a big company very successful i think vc funded so yeah so pipely.tech i think it's here to stay and pipely is the name of the repo whichever you go to it will just redirect you so the changelog pipely or the changelog pipetream there's a redirect and now we're having we had two contributions if you go to the roadmap the first one was make it easy to develop locally pull request seven from matt johnson that took a while to write some docker files explain how the pieces fit together there's a readme so if i go to pipely that we have docs which we didn't have before local dev so all of this explains what we're testing how we're testing quite a few things there so if you wanted to try pipely running it locally there's a doc that explains all of it so thank you matt for this contribution this one's great and i'm sure that we will build on top of it so matt did he do this himself and just document as he went or did he like do you know how he went about this so um i think there were moments when we got together so we had um okay let's go pipely.tech pipely.tech has the whole story there's no more three mages or three wise men it's just a world so the image has changed okay but we had let's build a cdn part two with matt nice and um james so they're there and um we'll link to the video so you can go and watch it and make it easy to develop locally so this was kind of like a follow-up to that so matt did a bunch of things if you go to pipely.tech you can read the whole story right now it's the second article let's build a cdn part two and this one make it easy to develop locally is the first one so in preparation for that the mat had to do a bunch of work right to understand how the pieces work what they are you know try running it locally and he cleaned all of those notes up and he contributed them to the repo so if anyone else wants to try this now they can now that's there so let us know what you think so that was one the second contribution which was completely unexpected is resolving the varnish tls issue so this was a big issue this was a big issue and we went deep so nabeel suleiman uh he's um what he's someone that you may have remember you may remember from a ship it episode we talked about the case cert it was a simpler alternative to cert manager that nabeel wrote because cert manager was too complex and i forget which episode exactly it was but you can go and look it up so he heard us talk about the issues that we had when it comes to varnish connecting to tls backends or tls origins and he wrote something that solves the problem it's called tls exterminator and now pipely is using tls exterminator to connect to origins that require tls termination how does it work in a nutshell we now spin two processes we now spin varnish and tls exterminator varnish connects to tls exterminator which then that proxies requests to https backends and that does the tls termination and all of that so with that we can now if i go back to actually i was here with that we can now add feedback ends now these urls they have https we could disable it we can go via http as well this is something to discuss that we want to disable it i think we should keep https on and if we want to keep https on we need a component that terminates tls between Varnish and the origin.

1:04:10So keep TLS on? Yeah. All right. So we're keeping TLS on. Great. Because again, HTTP currently is available, but I think we should disable that. So it's HTTPS only. We learned quite a bit with Nabil about why Varnish doesn't have support for TLS. So if we go to Varnish cache, why no SSL? There is a page on Varnish that talks about why SSL was not implemented. And you may be thinking, who wrote this? Poole Henning. If anyone doesn't know who Poole Henning is, let's look him up. This was 2011, by the way. TLDR, before we move on, it's OpenSSL is too complex. And when it comes to the implementation, if this was implemented in Varnish, which would have complicated the code significantly, the ssl proxying would have required a separate process and which is basically what tls exterminator is a separate process right the difference is that not only would have varnish been more complicated it would have been slower right so like the whole code that does with ssl would have slowed down varnish and that makes a lot of sense so who is pool hennig pool hennig camp uh he's a danish computer developer and he's known for work on previous projects such as freebsd and varnish so he's the guy that you can thank for freebsd he had a significant contribution and he is the top contributor on varnish some would say varnish is his idea but what's really surprising so let's go to pool healing again and he has not a freebsd there's that okay so we'll go through github phk.freebsd.dk so freebsd.dk apparently is his his tld and he just has a subdomain i think that's really cool so apparently varnish has a more license i had no idea about that and he's very transparent about the accounting like he he runs a lot of the software behind the varnish like docs and a couple of other things and he's very transparent about how he spends his time and how much he charges for it and like how like i was fascinated as an open source project how transparent it is and who contributes the most and things like that so popping the stack and just keeping going through who he is so free bsd apparently md5 crypt jails nano kernels time counters and the bike shed we invented the bike shed concept yep he's the guy behind the so and look at this when you refresh it the color changes colors so bikesh.org bikesh.org is what we're looking at pool heading camp has to come on and change a lot i think he does i think he does and the the pipely connection is just too strong to ignore so it's so strong all right so that explains why varnish doesn't have ssl and varnish enterprise does right so there's like the whole um commercial aspect but varnish open source does not have ssl and there's a couple of ways to solve it um and we may have talked about this on a recording that's not public yet so with nabil so we will wait for that to to to land but what does this mean in practice and this is where i go to the terminal so we're looking at pipely everything has been merged and anyone can follow along and we'll do the same here okay so let's do this alias j is for just right so just is something that i love and i think i mentioned about it just do it right it was like one of the that's right since kaisen 16 i think not the last one the one before last so there's a bunch of recipes that people can run and just debug is the one that we'll look at now this is in the context of pipely so pipely as you download it right now today this is what it has so what it does behind the scenes it's using it's using dagger and the reason why it's using dagger is because it needs to create a specific environment with different tools and it has to wire everything together so we can use in this case we're using dagger to publish package the container and publish the container even deploy the container so deploys our thing now we have deploys wired up so any any commit to pipely will go out and it will deploy a pipely application and we'll see that in a minute but now I just want to look at debug so what debug does it adds some extra tools on top of the application container so what are the tools let's just open it up and just have a quick look at what debug does so debug actually I forget it's not here it's in dagger main dot go so debug so for example we get curl on top of the application container which has just varnish t marks htop neo vim HTTP stat Sasquatch it's an interesting utility let's like watch with some extra features go top and oha and oha you will remember and then just i'll just obviously so it's just a way to interactively debug the container and try a few things out without polluting your system i think that's that's the key takeaway there so let's just let's just run that let's run debug and the the terminal function it's what puts us in that container so i ran the command and right now i'm in a container and i have a bunch of toolings available to me so what are the toolings if i do just again just is there i have a couple of commands to run i could run these things locally but really i just want all that to be wrapped because typos and a couple of a couple of other things so what would you like me to run first just backends just backends the first command so let's see what just backends does so it just wraps varnish adm back end dot list because varnish isn't running there's no backends to list what would be a back end back end would be for example the changelog origin the changelog application a back end would be the feeds origin or the assets origin so this is where backends get plugged into varnish and varnish provides caching for those backends cool so how do we start varnish let's see if jared up look at that see that's why we have something like this so just up boom there it is uh it's tmox is a terminal in the terminal so there's quite a few things there so uh just backends there's nothing there and if i do just uh check that's the one it does the first request uh it fails and the second one we can see we got a 200 right again it's really fast and again this is messed up all right it'll have to be horizontally i'm sorry it will have to be horizontally so just check it won't fit a lot but there you go there it is we can see http 200 okay and we can see that the request came from we got a hit it's a second hit from local so this came from varnish cool and if i do we do just just backends we see the two backends which are healthy cool what other commands should we run let's do bench cdn i think that's that's where actually bench origin so bench origin and you will recognize this this is going to oha and we are benchmarking that's beautiful yeah we're bench it's not as good as it runs locally there's a bit more detail but it's it's it's pretty decent i have to say so we have just benchmarked the change what we're doing here so this runs locally the benchmark runs locally but we are benchmarking the change log origin application which is production right now this is production okay yeah so we're benchmarking production how many requests per second 93.7 93 so about 90 requests per second now i'm in london this is actually split between new jersey and ashburn virginia so there's two data centers can go to either one it goes through the edge and then eventually connects there which then it has to connect to um i think the database it hits the database so 90 requests per second not great but only the cdn goes to the application directly so let's bench the cdn and we are sending a hundred thousand requests per second sorry a hundred thousand requests not per second a hundred thousand requests and let's see how long it takes so that took just under 10 seconds and we completed 10 000 requests per second 10 000 requests per second so the cdn we can see it's doing its job i'm connecting to it locally the the latency is low and this is our changelog.com cdn all right so now let's benchmark let's go to cdn2 cdn2 is pipely deployed on fly that now proxies to the origin so this is the new pipely that we're setting up and i think we already had this but how does it behave with the tls proxying with all of that right now we have all those things in place and we're almost complete remember we had about i think 10 000 11 000 something like that this one has 4 000 only so it's slightly slower it's going to cdn2.changer.com now the application itself has a shared cpu only 256 gigs of ram so it's like the the the smallest lowest cheapest cdn instance sorry it's a cheapest fly.io instance application is that that we can run so we could make it quicker we could make it bigger but that's not what i would like to show what i would like to show is if we benchmark varnish directly 73 000 73 000 requests per second and actually it's quicker it's 132 000 the problem is the benchmark we're only sending a thousand requests so let's just make it a little bit more let's just send a bit more let's send uh varnish let's send to it let's go via HTTP one one just needs a couple of things and let's go let's go a million so let's just go a bit more so let's benchmark varnish I messed something up let's see what it mess up bench that's one one there we go I just made a typo all right let's benchmark varnish so we are sending it a million requests per second where is this running everything is running locally it's running inside of dagger main request total how many requests total i think a million request total you said right is that a million request total we're sending a million oh i made a typo actually 10 million we're sending 10 million requests to it so let's see how does it behave exactly oops will we send it 10 million requests and we are more than halfway there so if i do if i go to this instance remember the same pop os instance and if i run uh btop i can see how this instance like what's happening here you can see the cpus there's a lot of red so this is now cpu bound actually and everything is local so there's no network because it happens in the same container in the same namespace same everything which means that this is really as fast as you get it and there's our result that's how many requests per second varnish can serve 211 000 two hundred eleven thousand local so varnish isn't slow it's caching well we can look at the distribution and because we're right there where varnish is there is tls exterminator that needs to talk to which terminates tls right so that's an external process which that connects to the origin and it can connect to multiple origins so right now we have only changelog configured but we'll have feeds we'll have a couple more this will run next to varnish and i think the pieces are starting to come together any thoughts this is cool man would we like and here's a question would we like to scale those instances up to see how much faster they will go if we provide bigger instances well what are we getting right now against our so current setup so our current setup which uses fastly we're getting yes between 10 and 11 000 requests per second so we're about halfway there we're about halfway there yes we're the cheapest smallest instance we're about 4 000 there you're saying that's all fast can do is 10 to how many thousand 10 000 it was about 10 to 11 000 so there's a couple of things that play this is like the the pop which is closest to me i have seen it go faster so i've done a couple of other benchmarks sometimes it goes to 16 000 17 000 so it can go faster um i think it just depends on network conditions load on their system but we are sharing right network with everybody but if i can push 11 000 requests per second that's a lot of requests per second by the way i think yeah it doesn't matter like it's 4 000 yeah so how fast is it enough if we were to do here you can see that right now my download i'm downloading 1.27 gigabits per second and my my network connection goes more my network connection goes all the way to two gigabits okay right so right now i'm about 10 000 11 000 so basically fastly is limiting my one connection i mean i say one connection one ip to about one point something gigabits per second and maybe we could benchmark it elsewhere but the point is there's you don't want like one user to use all your available bandwidth right so you need to apply some throttling so let me show you something interesting if i for example uh let's just do bench so i do just bench and if i do bench let's do remember bunny we have bunny changelog.com remember that the other cdn this how the other cdn behaves 1700 so let's go uh two and i would like to go let's go 100 000 requests per second let's see how that behaves 100 000 requests total 10 000 requests totals you keep saying per second sorry you do that's what adam was trying to fix earlier too yeah yeah sorry 100 100 000 requests we're sending a hundred thousand requests and we want to see how many can he serve per second and what we see here is that it stopped at 2000 at about just about 3000 requests they block you exactly they throttle and this was a surprise this was a surprise right so they have some sort of protection because i could be ddosing them sure imagine if it would be like a hundred of us doing the same thing yeah i mean we'd be sending hundreds of gigabytes and they were just sorry hundreds of gigabits they have to consume that bandwidth too like as a exactly structure exactly yeah okay so as you can see i mean i'm not liking this behavior i mean i can't benchmark it so from a benchmarking now it's resumed see so there must be stopped again and then it stopped exactly so it was just about 100 requests it led through so it's just blocking me and then letting more are you doing this via an api key how do you authenticate this uh is this just a i'm not i'm just hitting it as public like anyone okay i was gonna say if you can do this via an authenticated way then you can always just like pass a benchmark flag or something to get past this maybe but so for us i'm just hypothesizing how i would build it if i was building it you know yeah i would allow somebody to benchmark my system because there's gonna be times you want to benchmark your system yeah i would definitely look into that but i didn't have to do any such thing for fastly or for fly i could just send that a good thing though i wonder if That's a good thing because they're just like letting anybody just like benchmark them.

1:20:33You just sent 10 million requests to them. If they can handle it. I did, yeah. If they can handle it, yeah. Yeah, I did. You didn't just break a sweat. You almost sent 10 million per second. Yeah. That'll be correct. No, no, no, no. That's too much. I need more computers. I need my Windows machine for that. That's right. Two of everything is not enough in that case. Yeah. So my question, brass tacks, is pipe leave fast enough? That's the question. Correct. So let's scale it up. I feel like 4 ,000 requests a second versus the 10 to 11 ,000 you get on Fastly. Is that going to noticeably impact anybody?

1:21:08And I would assume the answer is no. Well, he's scaling that machine right now to test it out. I am. I know he is. I'm doing it. He's always scaling stuff. Let's pause for one second. Remember earlier in the show when we were talking about flying being down and stuff like that? Right. That wasn't hate. That was just facts. Okay. What he's doing right now is in the moment of having a conversation is essentially upgrading that machine to be more performant. Taking it from a cheap box to a slightly more expensive box. Yeah. And this is all via the fly command line. So cool. It is the coolest tech, man.

1:21:42They really are doing some cool stuff there. I love it. Yeah. I'm just wondering if we were to replace Fastly with Pipely, how well would we behave? Would we get the same thing? And how does it compare? But for that, I promise one more thing. So I'm going to deliver on one more thing, right? All right. Did you notice anything different about my setup? Like in here in the terminal? No, no, no. Like looking at my camera, did you notice anything different? It's super black behind you. Are you in a whole different space? Well, it was black before. Yeah, it was always black. Better camera? Yeah. Well, there was the black, it had some more detail, right?

1:22:19So before the black wasn't quite as dark. Okay. oh you just green screen yourself not quite all right so what's going on right now so something just went behind your head and it looks like grafana yes some sort of dashboard is behind your head gearhard yeah so that's one of the early birthday presents which i couldn't wait to open yeah your birthday is coming right up isn't it it is yeah it is i think by the time this will this will be out it'll be out uh so yeah if you're listening to this find gerhard tell him happy birthday thank you i appreciate that so um i always wanted to have a big ass monitor like really really big so big a bam as they call it a bab there you go i always wanted to have a bam bam bam he's got one and that's what happened behind me like the whole screen like the whole background is actually now a one giant screen this is a real screen back there yeah is it a tv screen or is it a tv screen it's a bam it's a bam tell us more about this yeah i've heard of this so what is it it's a it's a samsung s95d okay and it's a 65 inch tv so it's big and what it means is that i can talk to anyone and i can see exactly what's happening um across all every infrastructure right now i have the changelog infrastructure running there and do you see those spikes right there do you know what that spike is you just now yeah exactly that's me just now i did that i created that spike he's so proud of himself i did that spike yeah yeah you did so that spike right there is the benchmark that went directly to fly now on the left hand side as you as you look at it that's the all the metrics coming from the fly application our fly changelog application on the right it's all a honeycomb and because it's a bit blurry you can't see the details which is exactly what we would want right we would don't want to advertise all the details but really what's interesting is the shape of it so honeycomb i can't figure out how to automatically refresh grafana in fly has that capability so i just need to manually refresh it i have to click the refresh so uh let me do that he's there you go refresh he's leaning over leaning over i'm hitting refresh and then you should see that other half refresh right like half of the background refreshed yes and actually it's the same time stamp so i need to go the last 24 hours there you go now we're looking at the last 24 hours so do you see those spikes there yes those spikes is the benchmark which i did against fastly against our cdn so you can see that we never hit those levels under normal operating conditions right that's like yeah 100x what we normally operate so maybe being able to serve 10 000 requests per second doesn't make that much difference since really, we never hit those, those levels.

1:25:32I feel like we've gone to Target, the three of us, y 'all took me to the toy department and you said, pick a toy. I chose my toy. We went to the checkout. We, Target is a popular store here, by the way, if you didn't know Target, uh, we checked, we checked out, we successfully paid, we've left, we've gone home and you've not given me my toy. right where is my toy well i can't get the monitor for you you need to get it for yourself so okay so is the toy so pipely if you go to cdn2.changedog.com it runs it now uses a component that terminates tls to origins and now we need to add more origins while it's half as fast as the current cdn is we know that it can sustain all the load that we need to replace our CDN.

1:26:29So the toy is, the toy will work. In terms of what comes next, we need to configure more origins. We need to get, for example, the feeds one, the assets one, and we need to scale the instances in such a way so they can handle the traffic. Right now, we only serve, or we only save the assets, or we only save the responses in the actual memory. So we need to configure disk. there's a couple more things a couple more knobs to configure but this is getting closer and closer and closer i feel like the real toy is the samsung s95d 65 inch oled hdr pro glare free with motion accelerator gerhard that sucker is expensive man well nice ebay half price that's what i say brand new so you just need to shop around what do you do what adam does you know okay do what adam does basically well i was gonna let you know you know my birthday is like well all right in case you're wondering mine sooner march 17th march 17th i wasn't really jealous of all your computers you were talking about earlier but now your screen is amazing holy cow yeah that is so all right so scaled up our pipely to well the performance x1 performance 1x it didn't work uh yeah it failed so maybe we're trying to scale too much let us down oh dang man it was cool until i didn't know i didn't know what would happen maybe we uh yeah let's see if you do fly ctl let me just do that let me go flyctl machines list and let's see what's going on uh live debugging why not so we see performance one two only two really but the rest could not be scaled and i don't know exactly why so let's just do that again let's do vm scale updating machine see this this other one just couldn't update it and i'm not sure why exactly that's the one in heathrow waiting for machine to be happy sorry to be healthy to become happy a healthy machine is a healthy machine indeed that's right so that's still waiting for machine okay so now it's moving to the next one so just how many instances are we running right now uh one two three four five six seven eight nine ten ten so there's ten of them in different regions around the world yeah and we've got two of the 10 upgraded to performance 1x the other ones are on shared cpu 1x exactly and which has also uh 10x the ram it looks like so the shared cpu is at 256 megabytes whereas the performance 1x is at 2048 exactly yeah so that's quite a scale yeah it's it's about 10x and i'm wondering if we do that 10x like how will it behave and what would that affect the bottom line of running pipely because near now 10x our costs probably because you're upgraded every instance around the world i'm not sure how much it changes the cost i mean we we can check it exactly to see how much that would cost and maybe we don't need 10 maybe we need just one per continent maybe that will be enough or one per like east coast west coast like this is like you remember like the old one so there's a couple of optimizations which we can we can change there so what was the question how much will it cost well i was just wondering how much extra it is we don't need to get the exact answer okay these are just concerns that i have as we move forward and then is 10 even the right number is a question i mean maybe it's smarter to have to leave it at the shared cpu 1x but have 30 of them versus 10 at the performance 1x for instance yeah i think so we can see that like the we went with the cheapest one right smallest one um so shared cpu 1x uh you get i think you get the bandwidth which depends on how much um how big the instances right you get like a fair share of the bandwidth so these instances they were costing like less about two dollars per month um just in um compute costs we went to performance 1x which is 31 so that is more than a 10x jump maybe if we went to a 4x sorry shared cpu 4x which is about eight that would have been a 4x yeah 4x and also like a more realistic more realistic upgrade but i i wanted to make sure that we get like um we get the higher tier ones performance 1x is like the lowest high tier one which means that you get a full core is not getting throttled and my assumption is you'll also get more bandwidth and that's what we're testing here if we go to like the next tier of instance which is like compute optimize in a way it's like a huge jump but does that translate to bandwidth performance so we're still going through that i mean we can try benchmarking it again to see how it behaves and the reason why you could benchmark it again is because um the pop the one in he throw has already scaled so let's see you know how this one compares we are pushing for 80 470 480 okay so I think we'll get a similar result I think I needed 10 million again for 80 I did I think a hundred thousand requests are total thousand hundred thousand requests in totally just to throw some load its way and see how that behaves and we're 4 ,000 so apparently scaling scaling up the instance did not increase the bandwidth interesting mm-hmm So the question would be, is this as much as we can get?

1:32:12And should we, could we go higher? I don't know. Is the limitation then network? Is that what we just resolved to them? Because CPU and other things didn't really influence it? RAM didn't influence it? Yeah. So I would ask, for example, Fly, like how do they allocate network bandwidth based on instance size? Like how do those limits work? Yeah, that's not clear. and so that was the one question and what i'm wondering is 4 000 enough right because we're looking at the graph we're seeing the spikes and apparently we we never even hit 4 000 requests per second on our existing cdn it means that the ceiling is lower but since we're never hitting that ceiling maybe that's okay not to mention that we've seen bunny for example this is a perspective which i haven't seen in bunny before where we can see the throttling kicking in right we can't even benchmark it properly because it throttles you much earlier and i looked through the config i went through the settings like cdns apparently they're not all configured the same which is why i was looking at varnish to see what can varnish do like where exactly is this bottleneck coming from and are we okay with the ceiling is it necessary to have this um this throttling in place for who for i guess just the system the uptime of the system the vendors yeah it would make sense for me for us like i mean we would be at least temporarily not a pipely would not have a lot of uses users i would say like we would deploy pipely on prem basically like it would not be a service we're consuming pipely would be software we deploy for us to use and so do we really need throttling if we're our own user and we control our systems oh i see what you mean you see what i'm saying like yeah because bunny has it probably is a safeguard because they're public whereas pipely would be deployed for us in our use case right so we don't need throttling no deployed for us but it'd be hit by randos around the world true so we can get dosed yeah we could that is that is a real possibility you send us 5 000 requests a second and we're dothed to one pop at least one yeah yeah exactly yeah so i think some form of rudimentary throttling makes a lot of sense i don't think it would add very much in terms of software on our side to say you know you can make it very rudimentary this ip can only have so many requests a second done i think you're at least avoiding that low-hanging fruit i haven't looked into that but having this discussion is valuable i mean this is why you know like i don't think we can't have the toy because we're still debating what the toy should be and how it should behave but it's exactly we're building our toy and i think this just makes it more real because these are the steps that we would go through before we take this toy into production and i think this is the perspective which is valuable to have right like the level of care and attention and detail that we go through to make sure that what we put out there will behave correctly and the comparison that we have right now is fastly which you know from some perspectives it behaves really well i mean we're seeing performance is amazing caching not so good but again it makes sense why they you know don't keep content in memory as for example we would because we would optimize for that which means that because we optimize for that we want to store as much of it in memory as possible memory that we pay for or right no discs that we pay for wherever they may be and then i think this will also um we'll have questions about like how should we size those instances we just heard that maybe the performance 1x maybe it's a bit too expensive because we need to run a bunch of them and how many if you remember first time we were running 16 maybe that's a bit too many maybe 10 is a better number but even that might be might be too much now if we're looking at the cost right we're paying 30 dollars per instance and if we have um 10 of those we would be paying 300 dollars per month for the compute i think i think that's okay i think i think that's not crazy in terms of cost let me ask you a question maybe this is a stupid question but no let's ask it anyways we want to store are we designing the system to be memory heavy where we have terabytes of memory available to the system so we can store all of our data in memory i don't think so on disk yeah have lots of memory available if we need it so i think that we need both i think that the data which is hot should reside in memory and think about how zfs works the file system and when you have an arc so this would be exactly that yeah we would want to store the most often access data in memory and the one which is least accessed on disk so i think we need both because the memory we can scale it i mean if we go back to two gigabytes of memory right for ten dollars let's say we go we keep the 1x we can get eight gigabytes of memory that doesn't seem a lot of memory for example i wouldn't know and this is like where the cache statistics would come in handy how much data do we frequently serve and i know that we have the peaks right when we release something there's like a bulk of content that we serve often how much is the bulk of how much is the hot content i don't have an answer to that but all these things are getting us closer to those concerns shall i say that the system will need to take care of honestly i don't think that we should give it more than for example 16 gigs per instance and even that might be a bit big and i'm wondering whether all regions should have the same configuration and i'm thinking no because maybe in south america and i know this for a fact there's less traffic than for example in north america and maybe oceania i'm sorry asia let's go with asia again it's less traffic than we have in north america and even europe so then i think the same configuration across all regions doesn't make sense but knowing how much data is hot i think i think that's something important how would we know that just based on like stats to the direct stats itself yeah stats from the cache to see how much of those like how much cash is being used and are there any configurations and i haven't even looked into this are there any configuration in terms of evictions like how frequently should we automatically drop content i think this is where our cache hit ratio will come into play right so if you don't store enough of it in memory you will have a lower cache hit ratio while if you store too much maybe you're being wasteful i mean having a high cache ratio cash hit ratio while a lot of that data is infrequently used you're paying for memory that you don't need the other thing the other question which i have is are the nvme disks fast enough and if we think about netflix netflix does the same thing right they put their those big servers in isps they cache the content on those big servers so that they can deliver them really quickly to customers wherever they may be right we're not going to go there right so so this is not that uh but that's one pattern that they apply because they realize the importance of you know having lots of content close to users memory is not big enough you need disks again we're not there we don't have that problem well we're getting there i think we have some decisions to make as we go i think roughly speaking the dog hunts i think 4 000 requests per second well managed will be fine yeah and i think we'll find out otherwise and be able to scale one way or the other um around such issues what else is left in pipely's roadmap you know as we look towards the future now because yeah let's do next steps so i think that now we are finding the place where we can add feeds back end feed feeds back end and we also need the um static assets one so i would add both when we add them we need to figure out do we store all that in memory and i think the answer is no because especially static assets they'll use a lot of memory but maybe disk and i think we should we should look into that can we configure different backends like how does that work we're basically getting like to the hard part of configuring varnish for our various backends and each back end needs to have a different behavior i think so that's something to look into logs sending logs to honeycomb i think that is a much easier problem to solve because we would be using vector and now we have the building blocks which we have the first sidecar process if you want to think about that if you want to think about it like that which means that we have there's varnish and there's a couple of other smaller processes that support it we have tls terminator that terminates tls to origins to backends the second one would be this um in my mind it will be vector.dev which is what we would use for these logs so vector.dev would get the logs from varnish and send them to honeycomb so it's an integration which i've used before i know how it works it's very performant it's fairly easy to configure and then we'd have another helping process that would work in combination with varnish to accomplish a certain task and honeycomb and s3 like all those it supports multiple sinks so collecting the logs on one side and just sending them like in multiple sinks that is very straightforward because it just handles all that itself and then really the last the last hard bit is purge across all application instances and i think that one is like maybe a step too far to think about it now but i think the way we so first of all now we are automatically so we have an image to publish we are deploying the application automatically through ci that's like some plumbing that you want to have in place we have support for tls backends and that was an important one especially when it comes to other origins right because let's say if we are running in fly we can connect we can use the private network to connect to the changelog instance but for external origins like feeds we would need to go to http because we didn't have https so now we have https i think that's that's also like an important building block and now we're hitting the these there's like benchmarking i won't say we got sidetracked by it but i think it's like something worth considering because you may end up building something that won't work we won't be able to use this to replace our current cdn and the goal is to be able to say with confidence that pipely is able to do the work that currently our cdn is doing and what does that mean from a configuration perspective from resources perspective i think everything adds up and it it feels like it feels like we're more than halfway there for real i don't mean like will this work no no we're more than halfway there to replace fastly with pipely for us all right take us to the promised land here give us that toy it won't be christmas it will be before christmas adam when is your birthday march march okay oh that's too soon that's too soon jared yours is july okay all i want for my birthday is pipely and a Samsung S95D.

1:44:02All right. Well, I behaved very well, I think. And my wife must love me very much because it was, yeah, it was a present from her to me. So nice. Yeah. Very nice. So she loves the nerd in me. She has to, she has no choice. They come as a package. If she doesn't love the nerd, I mean, what's left? Well, Well, that's what you see, Jared. But this is on the show for that. Don't answer that. Well, that's cool. I love this exploration, though. I like that there's a possibility to run your own thing like this, to configure it to the way we want to. I mean, to zoom out, the challenge has been that it's been hard to configure Fastly, not as a CDN, but as a CDN we need.

1:44:51Our particular use case is not that Fastly is not good as a CDN. It's just it has not been highly configurable by us. It's been challenging over the years, mainly just because they're, I think, designed for different customer types. We're a different customer type. And we've been holding it not so much wrong, but it's just been a square peg around a hole kind of thing where it's not perfect for our particular CDN needs. I think we've had lots of cache misses over the years. Like, why is our stuff not cached? Yeah. It seems like it should be. We're not prioritized as a thing to serve because of the way the system works, and that's just it.

1:45:32And we're designing something that serves that kind of system, where it serves the data, holds more memory, has more available to it. It is not a mess. That's right. Which I think is cool. Very cool. Man, this tooling you built is so cool, man. I can't believe how cool this stuff is that you built. It's really awesome. thank you it's coming together it's uh i love the tv too yeah it is like a well-rounded experience right so the idea is to be able to have the tv on now it's a bit bright and it's it's it's running a little bit hot i can i can feel it it's it's not it's not uh uh winter yet i don't need heating in the office uh well it's like the end of the winter but um i'm i'm able to see a lot of metrics i think that's something that um i always loved to be able to see how things behave and when they misbehave to be able to see and understand at a glance what is wrong are getting ddos am i running out of memory like which instances is is problematic and i think this is just like a starting point i literally threw the two dashboards that we have over there but i haven't optimized them in in any way and i think having something like this just makes it in a more i don't like breathing breathing, living system.

1:46:49Well, yeah, you can see in real time, like what's happening. The metrics is the life, you know, it's each, you know, organ, so to speak. It's like your own knock. Yeah. Yeah. It is super cool. Super cool. I am jealous. I want one. I inspired you. The Christmas is far away, right? People can save. I know that we did. And it's been in the making for a long time. So before before i could get this i had to get a system that is able to power it that's what pop os does one of the things which it does you know it has a gpu right which is powerful enough to be able to to power it i have another monitor here which does like screaming rings so i can change things here and set things up a system that's able to be on and that it's not too loud that was like another consideration a black wall so it blends nicely it was like so many things like years in the making as pipely is years in the making that's right and it'll be just as beautiful is that i would love a uh tour a home lab tour i would love that coming soon to well i'm working towards that uh but i can i can show you one more thing which was not planned i know we're a bit over time but if you want to see one more thing i'll show you my m25 every year i basically uh take one of these machines online and i have like a 24 25 so this is this year this is the machine that that came online it's a it's running through nas as you can see it's an i9 990k it has 128 gigs of ram so it's like fully maxed out basically and this is how i'll just move that screen a little bit uh this is um you can see all the storage has two pools it has an ssd pool and an hdd pool um so spinning discs and some slower ssds um they're the evo i think 870 and um it's something that you need to have in place to be able to have decent storage between linux and windows and mac and everything to just work so that was one of the projects and i didn't have time to talk about it maybe maybe next time i know that you are a true nas user adam zfs and like all like all that stuff so yeah several polls several things similar to this slightly beefier machine it's a a xeon processor and it's a 4210 i want to say silver yeah i think there's a hundred and some gigs of ram i want to say 192 maybe 128 okay it's something like that it's not 256 i know that for sure right i don't have a need for it i mean it's it's nice it's a tinker's dream to have lots of ram and a zfs system but i just don't need it it's just you know i caught myself just wanting to have it just to have it and i'm like yeah that doesn't make any sense you know like you spend all the money on the ram just spending the discs instead yeah you know because discs are expensive yeah for video storage you know something that that you you would need be sure to do editing and especially if you edit it from multiple machines you need like a 10 fast network a couple of other things but yeah my home lab's suffering right now i don't have 10 gigabit everywhere i do have it in the network i just don't have it everywhere um so i'm i'm in the process of fixing that there's some some slight life updates for me that will make it more important i should just say i see i had a flood in the studio and oh wow okay i don't think I could stay here anymore.

1:50:19Let's just say I got to go home. So I'm turning my home office into a true home lab and work lab. And that's in the making. So techno team, it's a bummer. I need more techno team. Yeah. You know, I mean, I'll be close to the things I play with more frequently. I feel like I've always been like two location and it's been challenging. Cause like right now I can't access true NAS. It's at home. I can't access that windows PC. It's at home. It's in the home lab, you know? Yeah. And I've just sort of like stripped away more and more here to the point. point that it's like it doesn't make sense to stay here any longer well your background will change it's a pretty cool background that you have yeah that's the thing i gotta make sure it's you know video ready and that's you know uh i got a month to do that basically well and i know we went a bit long i know we covered a lot of stuff i dig this i love it i'm glad you showed this i would love on make it work do you mind if i promote that no no go for it go for it yeah go for it i would love to see a tour of whatever you can share it could just be iphone it could be low produced I don't really care.

1:51:16I just want to see what you're doing. Cause you know, what I love talking to Tim about in particular, one, I like him as a human being. He's so cool. And I truly think we're not just friends on the podcast, but I think, you know, if he wasn't 2000 miles away, I would hang out with him and spend time. Same with you. And you're a lot more than 2000 miles away. Yeah. But I love, there's not many geeks I can meet that like nerd out on hardware. Like you do. And like he does. and a couple others do out there that are friendly in the world, like Tom Lawrence. We met him years ago at a Microsoft something or other, I think, in New York.

1:51:54I haven't reached out since then. He's become more and more famous since then. So now I just watch him on YouTube, you know, and I appreciate his takes and stuff like that. But there's not a lot of geeky nerds who nerd out in hardware, like for no reason like we do. Like we, we build things we, we want to need. And so we make a need for it. You know what I mean? Yeah. Maybe there's some true need, but you're like, you justify it. Like this TV behind you. Cause like, wow. Why not have a knock? Like Jared said, when I have this, you know, big thing behind you and not let it be a green screen, let it be a real thing.

1:52:27Yeah. Just cause, you know, so make it work.fm to not hide the URL. Dot tv. That's what I would say. Dot tv. Dot tv. That's all new. By the way. That, oh gosh. Yeah. And that is, geez, I keep fat fingering it. I put a comma there instead. I haven't been there in a bit. To.fm. So this is still running from your home lab, right? No, actually this is running on fly and it has a CDN in front. Yeah. Because last time it was on your home lab stuff. It was, yeah. What was it? Jellyfin. Well, Jellyfin is still on my home lab. Jellyfin, like the media still served from the air because of the IGPU. but actually um like this one if we if we if i would just click on this one so there's a couple of things here and i'm logged in so it's obviously the the episode like the audio which is coming from transistor so there's like an embed and there's obviously the embed video and as a member when you sign in you get the whole thing this is served from the cdn directly so this is like the the cdn um content and there's also jellyfin so once you log in i see the quality for this one wasn't very good that's something that i'm still working on that's why i mentioned that i have to record my screen locally which is what i did for this uh because riverside is not great uh with screen recording they improved it but it's not there yet the quality is not as high distributed podcasting is so hard it really is like because you want to share that screen with us but then counting on Riverside to record it in a resolution that is good for long-term uses.

1:54:02Yeah. Yeah. Makeitwork.tv and.fm if you want to go the audio route only, but.tv is where you said to go, so go there instead. Yeah, man, I want a studio tour. I want something. Don't take six months. Do the simple version, Gerhard. Or just, hey, listen, we could just Zoom and just show me in the reel, you know? Okay. I mean, that's what you - We could just FaceTime. You could just show it to me. yeah we could definitely do that that much that's much easier it's the whole like backlog that i have to go through yes so that's i'm still working on that i i it took me such a long time to find like a good editor and i think i finally have him it took me at least four months five months of proper searching yeah to get someone that i'm also able to afford because this is still like everything self-funded uh but it works and uh my first i need to make make it work work before i can you know but um even like make it work on tv uh now there are subscribers and there's like all even like uh members like people can pay for it um so that's up and coming that was something i want to put to the spot we did talk about cpu for you so i'm hoping you're still excited i am i am yeah we're making steps so yeah maybe i'm keen to be part of that i just just did not have time between everything same you know we really just i've been focused on getting the agreement solid.

1:55:23I wanted to make a solid promise and have it be clear to folks. And so I think that's a simple thing, but understanding your terms between the people you're going to serve, I feel like you've got to examine that and have clarity there. Cool. Man, this has been a fun Kaizen, a deep Kaizen. If you've stuck around to now, holy moly, I'm not sure what's getting cut, but wow. You are a trooper, you're a super fan, and you should be a plus plus member. I'm not going to force you, but changelog.com slash plus plus. It is better. Bye friends. See you in the next one. See you in the next one. Kaizen. Kaizen.

1:55:59Kaizen. All right. That is changelog for this week. Thanks for Kaizen-ing with us. For the entire saga, head to changelog.com slash topic slash Kaizen. There you'll find all 18 Kaizen episodes. For your listening and now watching, pleasure. Thanks again to our sponsors of this episode. Please support them because they support us. Also, because they have awesome products and services. Thanks to Retool, to Sentry, and to Temporal. Links in the show notes. You know what to do. And thanks as always to Breakmaster Cylinder, the best beat freak in the entire universe. I think so. Do you? I'm sure you do.

1:56:40Next week on the Changelog, news on Monday, anti-res, yes, the creator of Redis on Wednesday. And on Friday, our first ever game of Friendly Feud. Have a great weekend, like and subscribe on YouTube if you dig it, and let's talk again real soon.

1:57:19We know your problems should be coding. And your deadline is pretty foreboding. Your ticket backlog is an actual problem. So why don't you go inside? No more listening to Change Lock and Friends. The Batman Chamber in Silicon Valley. No one gave a gag. We'll come to an end. But honestly, that will probably be our finale.

1:58:17We'll be right back. Change of your friends, change of your friends It's your favorite ever show Favorite ever show