Kaizen! Pipely is LIVE (Friends)

27:45It ran out of memory. Yes, of course. Thank you. I appreciate that answer. Someone's paying attention, but why did it run out of memory? Why do you think it ran out of memory? Because there wasn't enough memory. Right. I love some trolling. Seriously now. You're going to make me say it, right? Okay. So as more content gets cached in memory, the problem is there's like a configuration which I wish it was easier to make, but you have to manually adjust how much memory you give to Varnish out of the total memory available. So there's a dance that you need to make so that you know how much is enough so that when more memory gets allocated, the thing doesn't fall over.

28:27I wish this particular thing was easier, maybe something that we can improve, but for now, you have to fine-tune it and find, if you have four gigabytes of memory total, how much should Varnish be allowed to use? And if you think it's four gigabytes, it's way too much. So that's what we're going to do now. I'm going to switch to some live coding and show what happened.

28:51So actually, you know what, maybe let me try this. How's the font? Can everybody see the font? Can everybody see what's there? I'll make it a little bit bigger. Yeah, a little bit bigger, okay. So this is the change, and if I'm going to undo this, you'll see what it was before. so we I thought let me take responsibility of this I thought that 800 megabytes is going to be enough this was the case when the application had two gigabytes so in the instance had two gigabytes 800 megabytes of headroom was enough so the application wouldn't crash because of out of memory issues and apparently when you have four gigabytes 800 megabytes is not enough so what happened is 33 % 33 % should be enough for this to work and again you have to specify this explicitly I'm sure that we'll improve this at some point this is what this looks like so we're going to push this change into production live right now here it is, here's the change so we'll say increase varnish memory or limit let's do that, limit varnish memory to 66%.

30:0266%, right? Varnish memory can only use 66%. Okay, I committed. What's going on? Of course, I need to connect. Right, let's do that. Let's connect. Yeah, I am offline. Let's do this. Again, this is live. This is not recorded. It hasn't happened. Let's see how this... I normally record these things, but let's see what's going to happen. Good time to go live, Gerard. There you go. Let me just read this Bill Gates quote. Okay. You thought 800 megabytes ought to be enough. Yes. Bill Gates, 640K ought to be enough for anybody. Right, right. Apparently not. So, you know, it's not an unreasonable thing that you thought.

30:40Nope. Okay. All right. So what happened is we committed and we pushed this commit, this one commit, and to deploy something into production, all that we have to do is tag the repository. So tag and commit. So RC3 is the last one that went out. You can see when it went out, it was two days ago. We're going to do an RC4 now. And all we have to do is this. J tag. J stands for just. So I'll do just tag. Because I don't want to remember the command. It's quite long. So I'm going to do the SHA. The SHA in this case is going to be, sorry, tag. So V 1.0.0 RC? Four. Four, there you go. Nice. We'll just do head, right?

31:27of course is going to be head. And the discussion, this is the changelog discussion where you can basically listen about this thing. So this is us preparing Kaizen 20. Remember that GitHub discussion I talked about. That's what this is going to do. And I'm going to push it now. No, push. Git push. What happened? Undefined. That should have been fine. Looks like the connection. The connection dropped. Someone's blocking port 22. No, no, the connection dropped. Yeah, let me just go back. Let me read a Bill Gates quote. Yes, sure. Another one. Do we not have house in it? No, no, it's all good. Give me another Bill Gates quote.

32:06Let's get push again. Let me just make this. It's going to slow. There you go. That pushed. Cool. So the tag went out, and what we're going to see now is, actually, this one right here. It's going to go to the actions. So this is a live, this one right here, 1-0-0 RC4, and it's going to push this change into production across all instances. It's going to roll them live. We are deploying this. So why is this significant? Sorry. Let me set the record straight. Sure. Bill Gates in 1996 said, I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that, that a certain amount of memory is enough.

32:53Right. So I guess I take it back. Is that true? I don't know. I don't know. Is that factually correct? I scroll further. What is your source of information? Yes. So there you go. He may or may not have said that. So let's have. We're watching it roll out. I see, so now we're seeing like a live roll out. And we'll see, so publish and deploy tag, is doing all the validation, gone, gone, it's moving on, it's going through the changes. Errors, errors. Yeah, it's saying errors, but it's going to resolve itself. It just takes a while. So there you go. How long does it normally take, Gerhard? Well, last time, I think it took two and a half minutes.

33:28And you thought this was a good idea. Of course, why not? Look, we're already creating the new machines. Look at that. That's going. All right. So this is something that I think is important to get right early on. And that thing is, how long does it take you to push a change into production? So how long does it take you to make a change and see that change roll live? And this is something that we've been working on for quite a few years on this thing. we do push to production we own our production, we don't develop in production, that would be crazy even for us, but it's like all like this mentality of if I'm going to make a change, how long will it take me to see that change happen?

34:07And if you can shorten that time, you're in a good place 10 years ago it took 20 minutes or so It was long It was a long time. It was long enough that you go do something else And now it's two and a half minutes maybe. And this is a global CDN. So this is, we are pushing this change to all the instances of the global CDN that we run in, where do we run them? Let's have a look. So this is it. So we are Ashburn, Virginia. These are the instances two days ago. If I'm going to refresh this, you'll see the new instances come live. So that was two days ago. That was like the last deploy. Let's refresh.

34:43Deploying 57 now. That's the commit. We have new instances coming up. And you can see like we do blue-green. Of course we need to blue-green We need to deploy the new ones. The old ones are still there. Two of everything. It's an important rule. And yeah, this works fairly well. It will not have any downtime. And we can look at that in a minute. So Ashburn, Virginia, Chicago, Dallas, Texas, Santiago, Chile, San Jose, California, Heathrow, of course, London, Frankfurt, Sydney, Australia, Singapore, and Johannesburg. So these are all places where we deploy these instances. and they will accelerate all the content to our users.

35:23Does anybody out there have a deploy pipeline that runs faster than two and a half minutes?

35:32Does anyone have a deploy pipeline? Let's start there. I got a hand up. Two people, three people, four people. So I think we're winning. Five people, six people. Cool. Two and a half. Two and a half, nice. It only feels long when you're on stage. Yeah. Deploy now. Well, this is real. This is real. Like, not edited. This is what it feels like. Now, I don't think two and a half minutes is long. I think we can improve it. But you need to think about all the things need to happen behind the scenes, right? The allocation of resources, the health checking. Like, how do you know what you've put out there is correct?

36:05And you need to wait a while to make sure that the thing doesn't crash. That's why you need to wait at least 60 seconds before you can say, yep, this is good. You need to do a few health checks. because if the thing starts falling over, you don't want to leave that thing running in production, of course. Cool. So I think we're in a good place. The one thing which I wanted to show is if I come back here, can you see that? Is that graph looking good to you? Yeah, cool. So do you see this yellow line? This was the instance that crashed. So this one, if I, there we go, it's San Jose, California. For some reason, there is a lot of requests hitting this instance, and those requests they don't look like human requests so I think we may have some sorts of bot situation going on some sort of I know LLM trying to learn there's yeah there's like a lot of lot of requests and if I'm going to look at this one the CPU utilization this is the same instance right here the one in San Jose California I mean look just how abnormal this instance behaves and if I'm going to remove that you can see all the others so all the other instances the CPU usage is fine, but this one is spiking up to 100 % because it has a lot of requests.

37:15Now, obviously this is in front of the application, so five-minute load average. We can also see here San Jose, California. Yes? Can we see the endpoints that it's serving? The endpoints that it's serving? We can, yes. So let's do like a tiny reveal right now. Jared's like, can we do X? Of course we can. Of course we can, Jared. Of course, yeah. No, we can. We can definitely do this. Okay. So I'm going to switch to this view. And this is the last seven days. And I'm going to make this a little bit bigger for you to see. And can you see that all right? Okay. So you can see that we're here, right?

37:56July 26th. So the application went into production a few days ago. So, not all of it, not completely, but we started routing a bunch of traffic to the application to see how well it would handle our users. Is that the biggest thing we ever did? Almost. It's getting close. I'm still waiting for this. It's getting close. Okay. But this basically shows that we had many, many steps towards this moment. and now, together with you, we've shown you how we can update something that's running live that is serving, I mean, this doesn't sound like a lot of requests, like a thousand, but the granularity is 30 minutes.

38:39And we are sending a portion of the traffic and it's handling it pretty well. And we can see that the biggest users or like the most requests are coming from ORD. Chicago. Chicago, that's the one. Frankfurt next, London Heathrow, and it's not me running load tests in Singapore. So these are like the, and then San Jose, California right here. So we are running live traffic, not all of it, a portion of it, but we wanted to see, will this thing continue working well? Did you know that? No. I don't think so. Yeah. So technically we're not launching PipeDream today because we launched it on Thursday, just me and you.

39:18Exactly. Sorry, team. We didn't. We barely launched it. We launched one out of five. One out of five, yeah. Yeah, so roughly 20 % of requests go through our pipe dream at this point. So whenever you go to changelog.com, that's it. One out of five requests hit the new instance. And we were able to see how does it behave with 20 % of traffic. And it's working. No one complained. Adam didn't even notice. And that's like one of the best things, right, like to roll things out. You do it in such a way so that, I mean, it is a big thing to us and to people that understand and know what goes behind it, but to everyone else, did anything change?

40:01Because if you do this type of job correctly, all people see is maybe little improvements, and if they're not paying attention, even those they will miss. They think you're always just badass. Well, I think we are a good team.

40:18And the other thing which I want to mention is that, And what you see here, again, it feels very, it just happened, right? It's like a small thing. The work that went into it, it was months and months of preparation, months and months of discussion, people joining us, working with us. I want to thank James. I mean, he was the first one that has joined. Thank you, James. Thank you very much. James A. Rosen. Yeah, thank you, James. Thank you. Talking in the various issues, the first one that we had, basically being with us for almost two years now, discussing about problems that we thought were problems.

40:57At some point, we were questioning, like, are we maybe too strict? Are we too demanding out of this? And no, no, I mean, we want things to be better, and this is why we want things to be better. So James, thank you very much for all the conversations, keeping us, like, big picture, like, that's all objective perspective as well. That was so, so helpful. and then Matt, right? Matt Johnson, thank you very much for that. Thank you. Just, thank you. What a matter. Thank you. It was all about like VCL and going a bit deeper just to have like another perspective on VCL. I mean, he has a lot of experience in VCL and Varnish in general, but also documentation, also like a diligent approach to how should we do this so that it's easier for others?

41:42And that was like so great to have that help. Now, that was months and months and months of, I mean, again, all of us have full-time jobs. All of us do something else. But in our spare time, we find a bit of time to help others. And this was that. So thank you very much for that. Matt had a good idea last night. Can I pitch it to you? Of course. Yes, please. Yes. So my desire was a 20-line BCL. Yes. You gave me 1 ,000 lines. Mm-hmm. Matt says... We can count. You guys can just pull most of it out into an ink list. Yeah, that's cheating. but yes. I won't look. Yeah, of course. Great. Every time I go to the repo, I'll be like, oh, this is nice.

42:19We can give you 20 lines, Jerry. We can give you less than 20 lines, yes. I think that's a very good idea. Yeah, just look number one, like step number one. Yeah, exactly. We'll kaisen that, for sure. So, I wasn't prepared to do this, but let's do it. So, we have this is just, we have like a bunch of targets here. There's one that says how many lines? Oh my goodness. Right, so let's just run that live and see what happens. So, how many lines I'm going to tap this. So let's see how many lines we have. Okay. 961. Like, hang on. Let's see what's in Varnish. Hang on a second. Oh, no, we don't want that.

42:52This is not correct. We just want ours. We just want ours. Okay. Shall we change this live? All right. Let's see what's going on. What's going to happen. So why not? So we want only, uh, only VC. Actually, no, Varnish. No, we want inside VCL. So yeah, that's just like one change. So let's go just file. Do you want to do it? Nope. I feel like you're stepping in. All right. So let's see. How many lines? How many lines? There you go. So varnish. Let's do varnish. All right. Let's see that. Varnish, sorry, VCL. VCL. Okay. Let's see if that works. There you go. How many lines? There you go. These are the actual lines.

43:26All the lines. So 364, that's like the main one. And these are like the includes. 24 in this one and 106 in 15. Right. How many total? Less than 1 ,000. I think it's less than 1 ,000. 500? Yeah, ish. Getting to 500, between 450 and 500. A lot of these are just like static redirects. Any savants out there, do the math. Say again. Any savants out there, do the math. Look at that. 15. This is how we do it. 106 plus 24 plus 364. Look at that. 509. 509. 509 lines. So there we have our answer. Cool. So it's still good. Still good. All right. So. Do you want to thank Nabil? Yes, of course. Nabil. How could I forget the wheel?

44:09Of course. So one really important thing is that Varnish cannot terminate backends which have SSL in front. So if the backend is talking HTTPS, Varnish cannot use it out of the box. Varnish Enterprise and other products can, but the open source Varnish cannot. We did a bit of digging, and we realized that's how we learned about Pool Handing Camp, PHK for short, and he was always against including SSL anywhere near Varnish because it would complicate things too much. SSL is really, really complicated. So with Nabil's help, we wrote, I don't know, like he wrote 50, 60 lines of Go code that intercepts all the requests going to those backends, terminates SSL for Varnish, and presents the request unencrypted.

45:00A really simple, elegant solution, one of the almost like sidecars that sits next to Varnish and helps it terminate requests which need SSL. So I'm not sure if that's... Is that a new justice, Nabil? Genius. Absolutely genius. Let's hear it for him. Nabil. Good job, Nabil. And his SSL termination. Thank you. TLS exterminator. TLS exterminator. That's the one. That's a good name. Cool.

45:29Well, friends, it's all about faster builds. Teams with faster builds ship faster. and win over the competition. It's just science. And I'm here with Kyle Galbraith, co-founder and CEO of Depot. Okay. So Kyle, based on the premise that most teams want faster builds, that's probably a truth. If they're using CI providers, their stock configuration or GitHub actions, are they wrong? Are they not getting the fastest builds possible? I would take it a step further and say, if you're using any CI provider with just the basic things that they give you, which is if you think about a CI provider, it is in essence a lowest common denominator generic VM.

46:09And then you're left to your own devices to essentially configure that VM and configure your build pipeline, effectively pushing down to you, the developer, the responsibility of optimizing and making those builds fast, making them fast, making them secure, making them cost effective, like all pushed down to you. The problem with modern day CI providers is there's still a set of features and a set of capabilities that a CI provider could give a developer that makes their builds more performant out of the box, makes their builds more cost effective out of the box and more secure out of the box.

46:45I think a lot of folks adopt GitHub Actions for its ease of implementation and being close to where their source code already lives inside of GitHub. And they do care about build performance and they do put in the work to optimize those builds. But fundamentally CI providers today don't prioritize performance. Performance is not a top level entity inside of generic CI providers. Yes. Okay, friends, save your time, get faster builds with Depot, Docker builds, faster GitHub action runners, and distributed remote caching for Bazel, Go, Gradle, Turbo Repo, and more. Depot is on a mission to give you back your dev time and help you get faster build times with a one line code change.

47:26Learn more at depot.dev. Get started with a seven-day free trial. No credit card required. Again, depot.dev.

47:38We improved it. We're here. We did it. Kaizen 20. One more thing. Okay. This is too far out. This is too far out. Too far. Actually, no. We're good. Yeah, we're good. We're good. Okay. We're doing good. Anything else you want to cover before we do this? There's one more thing that we're going to do. one more thing Kaizen20 any thoughts any comments from the, I mean you're here do you have any questions, do you have any comments do you have any thoughts, anything we have a question, we're among friends can we talk about how you're using Dagger can we talk about how we're using Dagger alright, okay I can show you about that yeah, show them so there is one very tiny change that I have here, which is using dash dash cloud.

48:29This is an experimental feature that exists in the Dagger CLI. The reason why we do this is because we need a remote engine, a remote Dagger engine, so that everything that runs in the context of Dagger is quicker. Over Wi-Fi, it would be over, like my tethered connection, would be very, very slow. So how we use Dagger in this case, let's go here. We have a couple of commands. The one I think that's the most interesting one for example, for running tests. So let me show you what that looks like. I'm just doing just test. You can see there it does like the dash dash cloud. Let's see how fast it goes.

49:05It starts an engine in the cloud. It connects to it. So we need to set up Varnish. We need to connect Varnish to all the backends, the TLS exterminator. And we want to do it in a way that is as close to production as possible. So it's almost like we want to run the system as it runs in production, but we want to do it locally. Okay, in this case, it's a remote engine, but from the perspective of how it gets put together, it's literally just the same context of running containers. And because everything runs in containers, we get the exact reproducibility that we get in Fly. So we get the same configuration, the same Linux subsystem, the same kernel or like a very similar kernel.

49:49And what that means, I mean, in this case, it's even using Firecracker behind the scenes, so it's very close to Fly. so we are able to test the system most accurately and we're even able to run the system most accurately as it would do in production and that part is hard because whatever you do locally if I was running for example on the Mac everything would be different even if I would have a VM things would be different so it's like that containers the container image the interaction between all the components and when we ship something in production it's as close as possible to that image. Even down to the Go version.

50:22So right now, for example, what's happening here, we're pulling down some of it is cached. We're pulling down various dependencies and you can see they're all Linux ones. So Linux, Linux and again, a VM on a Mac, it gets you close, but it's not the same thing. You get subtle differences. I'm not sure if that answers your question. Yeah? Cool. Good use of Dagger. Any other questions before we, in the back, do the big thing? I think Nabil had one, was it? Yeah. Yeah.

50:56Does Varnish only cache to RAM or does it also use disk? So it can use disk as well, but we don't have disk configured. So the configuration is cache to RAM only. It is the fastest one. If we have disks, as you know, the problem is the host, they can't move around. So it's no longer stateless. That has certain challenges about, like, placement. but also disks tend to be a little bit slower. Not by much, but a little bit slower. As an optimization, it would be worth exploring that, especially with NVMe disks, which is what we would get in Fly. So that is something as a future improvement, but right now everything gets served from memory, which is exactly what happens in Fastly as well, because that's what gives you that highest performance.

51:39So follow up to that. our crash last night was because it needed more RAM than we allocated to Varnish. And in Varnish, there's no way to say, just use all the RAM available. So if you tell it that, it uses more than it has available because it tries to allocate more than you have. So you need to set an upper limit. It can't know how much the machine has? Well, it knows, but it doesn't do the allocation correctly. Is that a bug? I don't know whether it's a bug. It's just not working well with the system available to the host. Maybe, maybe. Varnish is how many years old? So 20-something. 20-something.

52:17There's no way that bug is a bug, right? It has to be a decision. Right, so I gained a lot of respect for memory specifically and memory problems because they're very, very hard. I spent maybe three years optimizing memory allocations on the RabbitMQ team for RabbitMQ in the context of Erlang. and I've learned that there are so many subtle differences between how memory gets allocated on different runtimes. In the case of Varnish, obviously it's C, so it's as efficient as it gets, but it's very difficult to know how much you should allocate ahead of time and what you should free, timing, like something has somewhere to give.

52:56And usually what you need to do is over-allocate. Basically, you give it more than it needs so that when you get these spikes, right, There's enough for it to spike so that it doesn't crash things over. Memory these days is cheap. So honestly, skimping on that is not worth it.

53:18And there's a couple of ways we can go around it, like disks is one of them, limiting what Varnish can do, but also seeing if the Varnish memory allocation can improve. Because I know what it takes, for example, what it took in RabbitMQ to do that. it took me years and I was like full time on that so it took a while including to understand what happens behind the scenes so you need to break it down in a way that you map and observe everything when it comes to memory allocation and then you realize what needs to be tuned to your use case that's the other thing because every context tends to be different so the allocations they are generic to make them configurable to you or specific to you so that they're optimized for your use case it takes a bit of understanding of what you actually need.

54:06Yeah. I'm probably reducing it down too much, but I feel like you could just say, this machine has four gigs. Just use three, and if you need more, you've got another gig in there. Right, so... Just don't crash. Yeah, true, yes. You know? Yeah. But my question, my follow-up, I guess, would be, assume that we figure that out, that little dance. Yep. What happens when our instances become overwhelmed? Varnish might not be crashing, but maybe it's slowing down, maybe the machine's bogged, can't do stuff. Is there an auto scale? I mean, the answer is more RAM on the fly instances or more fly instances in other regions or in the same region.

54:45That's right, yeah. So there is a big difference between regions, by the way. And this is something like, this is the sort of thing that you only realize once you start using it, okay? So this is, these are like the last 24 hours. I'm not going to refresh. It's a little bit outdated. I did it like you can see here. It was 10.42 a.m. It was this morning before we started recording. And what you can see here is this one, the San Jose, California, the SJC, is the one which shows the highest fluctuation. A lot of these, once they load up on memory, like Heathrow, they tend to be fairly stable. But then you have a few, for example, this one, Santiago, Chile.

55:20This is like 2.3 gigs. So this one is not using all the available memory. This next one in Sydney is 1.84. And you can see that the memory is decreasing because it doesn't need. I mean, this is something I'm not sure why does this, for example. I wouldn't expect this line to go down. I would expect it to stay stable. So I'm not quite sure what's happening here. But this one, the lowest one in Johannesburg, it's using 816 megabytes of RAM. So this shows you there is a difference between different regions and how many requests they need to serve. So what I'm thinking is we should optimize for the regions that we have, and the ones which are busy, we should give them bigger instances, maybe beefier instances, but others like Johannesburg doesn't need that much.

56:01So maybe we, and the problem is you can't use different scaling strategies for different nodes and they have multiple deployments so it complicates things a little bit. But this is a refinement. Yeah. I think we need more listeners in Johannesburg. I mean, what's up with that? Exactly. That's what I was thinking. That's what I was thinking. All right, there was one other question out there, I thought. It was back here. Yeah. I wanted to just put it around in my comparison to go back and see what the past looks like. Okay, yeah. Yes. I like it. Okay, so... The paths look like in Santa Claus. Right.

56:30I forgot about that. Thank you. Yeah, thank you. Thank you very much. So this is... We're looking at Honeycomb, and one of the integrations that we have in Pipedream is we are sending every single request to Honeycomb and some requests to S3. So, like, we see what's happening with these instances. So what that means is we are able to, for example, if I come back to the boards, if I come back to the boards again, Let's hope the tethering works. It will be a little bit slow. Okay, so let's go PipeDreamRequests, and I think maybe PipeDreamContent, but let's go Requests. We can see the get, okay, and now we can slice them and dice them.

57:12The 404s and the 500s. So maybe let's do gets. Okay, so let's go to the method, and the only thing that we need to do, so this is one hour ago. So these are all the requests in the last hour, and we can say, give me the URL, which is what we've been asking. So group by URL. And what this is going to show us is the get, but also the number of requests. So what you see here is that the most popular request is podcast feed, which got 203 requests in the last hour. And this is global. So we can do one more. We can say, let me remove that and let me remove request. And let me do data center. So we can see which data centers hit the most.

58:00So let's go that. And we can see that Frankfurt, 71 requests went to podcast feed. Chicago, the next one, 47. San Jose, California. So these are the most requested URLs. If we zoom out a little bit, let's look at the last seven days so we get like a bigger perspective. Add it to seven days. Okay, so you can see when it went live. So you can see that for some reason this one, I don't know who this person is, uploads 9 ,500 times. Shall we go and check who this is? I don't think you'll know who this is. Avatars. Somebody has a stalker. People. Let's see. Okay. So let's go to changelog.com. No, let's go.

58:41Where is it? Where was it? This one, this one. Okay. Copy that. Copy pasting. Hard. Uploads. And I think this will be CDN. CDN. There you go. So let's see who this is. oh yeah someone you have a stalker or a few who knows who is that guy yeah yeah you recognize him who's this other guy like 6qd who's so that's z4 or z4 z as you say it over here all right cool all right so let's see what's this next other person this is a fun side question right now Why not? There we go. It's Cable. Cable. There you go. He was in on it. Shall we keep going? Do you want to find who this person is? Let's see all the, yes.

59:26That's like the third most popular one, so we might as well. Come on, be Adam. Yeah, Adam, let's see. All right. Let's try that, uploads. There you go. So someone is loading maybe a JS Party page way too many times. That's possible. Oh yeah. But it will show up here, if it was the case. True. But then we have this randomness thing. Is dysfunctional doing something with something? Oh, on their website? Maybe. So, you know what we could do? We could do user agent. That's the user agent, and we'll see where these requests are coming from. It's gonna be a robot. Empty string. They don't wanna be known.

1:00:08Okay, shall we do... Masquerading and hiding. Let's get to the San Jose URLs. This is what we're trying to get, right? San Jose URLs. Okay, let's do that. So let's do, okay. So it's not even here in the list if you look at that. And look, this one has an empty string. So that's really interesting. We have somewhere an issue. So this needs to be, or no, hang on. This was like when we were doing the testing. So maybe that's one not so much practical AI. All right. Okay, so let's do server data center equals SJC. Yeah. Run query. And let's see, what do we get? Most popular in San Jose. How you can do all this slicing and dicing.

1:00:45Yeah, that's really cool, isn't it? It's so awesome. Look at that. Pocket casts. This MP3 was requested a lot. 428 times. Is this the last episode? Yeah, that one just went out yesterday, right? There you go. So it's just a popular episode in San Jose. That's it. That's better than I thought it was going to be. Which is a hacker. Yeah, no, this is good. Popular in Cali, not a problem. Look, overcast. Pocket casts and overcast and ten of pods. Yeah, they're just downloading stuff. This is good. This is good traffic. We love it. Yeah, nothing wrong with that. Cool. Okay, any other technical or otherwise questions from the audience?

1:01:20Comments, anything. Before Gerhard does his big reveal. All right. Everybody wants to see what you got, Gerhard. All right, let's do it. Cool. So, coming back here. It's the one more thing. This is the important one. No, that's something else. Cool. Flash something. All right, I flashed something, yes. We'll see it right there. So what we're going to do now, we are going to shift all the traffic to PipeDream. So all the production traffic is going to go to PipeDream. Shall we do it? Get your podcast. Of course we should do it. That's why we're here. All right. This is the moment. So all that we have to do, this is how simple it is.

1:02:04It's always DNS. No scripting. It is. No automation. It is DNS. It all comes down to that. Yes. We are going to delete these one by one. These are the A records which are pointing to. Pause. Let me get my own recording of you doing this. Go for it. Okay. Kaizen 20 in Denver. Let's do that. This is going to be anticlimactic because you're going to delete those and then we'll be like, they're deleted. Now, if this one goes down, so let's see if the system can handle all the loads, right? Like what can happen? So after this, the only requirement is we walk away, okay? We do this thing and we go for lunch or something like that.

1:02:40That's what's next is lunch, I think. All right, so boom, we have four more to go. Okay, so now we're - Three more to go, sorry. Yeah, after this one, three more to go. It's not deleting. It's not deleting, it's slow, right? It's my tether. Shout out to DNSimple, big fans. Thank you very much. One more, one more, cool. And the DNS is like one by one. So more and more requests are going to go to this one, which is the pipe dream. The pipe dream. The pipe dream, there you go. All right. So delete. So we're at one out of three. Now half our trash. Yeah, one out of three, yeah. Now half our traffic will be going to Pipe Green.

1:03:12Yep. Look at that. And now 100%. Now, we have one more. So we're 50-50. What's this one? That was it. That's CDN. That's CDN. We have to do the same thing for CDN as well. You've got to delete the CDN ones too. There you go. All right. So this is app requests now. Is this goodbye to Fastly? This is it. This is the farewell to Fastly? This is it. This is a triumphant moment in a morium at the same time. Only if it works. Only if it works, yeah. If not, we have to go crawling back to them. I may need to add these back. Like a dog or a dog. We still love you, Wesley. If this crashes and burns, we'll have to go back for sure.

1:03:45The good news is they have no idea what we're doing. Yeah, exactly. And this is not a live live show, so we're not streaming this. That's right. So it's okay. That's right. It's in this room. Amongst friends. If it doesn't work out, we edit this part out. So that's okay. He always says that we never do. Okay? Cool. We ship it all. There you go. One more to go. I think this is one and one more to go. Now, how long is it? We're at 60-second TTL. So this should be fast. 60-second TTL. This should be fast, yes. All right. If you can get your phone out and hit our website, please. Let us know how it goes, but that's it.

1:04:14We only have these two. Let's generate some traffic. Where's the real-time dashboard showing the activity? Real-time dashboard, of course. Let's do that. Let's do last 15 minutes. Let's see everything crash and burn. Oh, my gosh. No. What's the worst that could happen? Okay. The worst that could happen is all our fly-ins run out of memory. Well, there's no peaks and valleys. Let's go back to here. Let's go to the memory. There we go. Every one minute, let's maybe go to the last one hour. so let's see we had like a spike there but that was this is like 1220 so I think it's still good I think what we're going to see so two things we're going to see we're going to see let's come back here to the pipeline request we're going to see a change like these requests will start going up more and more traffic is going to start hitting look at that oh my goodness that's boom that was like the spike right there more requests getting resolved here status still 200 that's good.

1:05:05So that's all good. And 404 is 500. So things are looking good. Let's see what else do we have. Let's load a few more boards. And the opposite is obviously in here, in Fastly. Nice. So it kind of works still. It works on my machine. It's so fast. Fastly service stats. So these are Fastly service stats. The requests went up as well. I'm not sure what happened there. Why did they go up? Because I just told everybody to go to Right. I don't think we have that many people here, but still, cool. So, there's the requests. All good? Playing MP3s. Playing MP3s. Play the show. Let's do Pyedream service.

1:05:48So there we have a few again. The cache, we can see things going up here. So this is the last seven days. We'll need to zoom in a bit, so let's go 24 hours so that we see the spikes. There you go, that was the spike. Yeah. So. What is that spike? This one is more hits. That's good. More hits. We're getting more hits. That's good. More hits. So it will take time because the real question is, was it all worth it? And the answer to that is, well, do we fix that cash hit ratio? Let's check it out. That's the actual answer to the question. What's the 17 %? Are we going back to that? So let's go back to that.

1:06:2217 % cash hits. Last 24 hours. This is the homepage, right? That's the one that we're focusing at. Sure. So we're looking at the homepage. Last one day. We had 3 ,700 hits. Yes. And we had 33 misses. I think that's better. That's better. It is better. It is better. We Kaizen'd it. It's better. We did it. No way. No way. Do the math on that. Let's see. 3, 7, 1, 5 plus 33. That's 99.1 % hit rate. That's a lot of nines. Yeah. Two. To be exact. That's two nines. That's way more nines than we did. That's two nines. More that we had. There are some nines in there. It was like 70%. 17%, so this is much better.

1:07:01All right, so obviously this will take a while, right, for all the traffic to start shifting through. It is DNS, right, it's cached. But things, look at that, things are happening. Things are happening. So theoretically, 60 seconds later, those TLS records should expire. Exactly. And then require a new hit, which goes to a new route, which is through pipeline. So let's see this, dig change.com, and change.com right now, for me, it returns a single IP address. That's it. Bam. Because we're still in Ashburn. No, this is the Pipedream IP address. This is it. So this is one. And if I do changelog.com, this is the DNS that updates the quickest.

1:07:39That is the Google DNS. That's very fast. So Google DNS knows we're up. There's also, I use DNS checker. Let's see if it's still a thing. It's been a while since I've used DNS checker. A service that checks, tries to resolve the IP address from a couple of locations, more than a couple. Let's see. DNS checker. Let's go changelog.com. Try CDN. We'll do CDN as well. changelog.com. This is, I think, the important one. So all the DNS in San Francisco, it is. I'm going to make this a little bit bigger for everyone to see. It's a new IP address. We can see all the locations. So all of it is the new IPs.

1:08:21We don't see any of the old ones. The world knows about what we did. Yeah, I think so. They took notice. I think it's been good, I think. Yes, question? .

1:08:35That is a fly.io IP address, yeah. That is a fly.io IP address. Let me show you that so we have just IPs. We can see what IPs the CDN is using. And you will see that it's this IP address, which is a dedicated one. That's the IP that we're using. That's it. Should they clap now? Only if you want. You did a car. Thank you, thank you. Mind-blowing. We did a thing. We did a thing. Thank you. Thank you. Thank you. Because you were a part of this. And we did it with you and for you, and this was so good. Thank you very much. Thank you. So a couple of special things. We've already thanked our Pipely folks.

1:09:18Thanks to a couple of Denver liaisons. Dan Moore. Is Dan here? Hey, Dan. How are you? Dan. Thank you, Dan. And Kendall Miller. Did Kendall leave or is he still sticking around? He left. He was here earlier. Thank you to you two for helping us find this theater, for helping us connect with Nora again after all those years. We don't live here and so I had no idea what I was doing or who I was talking to. So always awesome to have locals and friends willing to help out and make it awesome. That's our show. One person didn't get thanked. Yes, Jason Oh yeah, Jason, where is Jason? Come up here, Jason Jason, come here, Jason's our editor He's behind the scenes, but he's very much part of this team, he doesn't get seen He gets mentioned a lot But critical, critical behind the scenes here at ChangeLog Thank you, Jason Thank you, Jason

1:10:18Alright, anything else, Gerhard? Thank you all, I really appreciate it Really appreciate it, thank you all Thank you.

1:10:28There you have it, our first ever Kaizen Live. But I'm pretty sure it will not be the last. You know an idea has legs when you're already brainstorming version 2, before version 1 is even out there. And we certainly were. Stay tuned for more. This particular episode is better on YouTube, and we have more videos from the Oriental Theater coming soon, including BMC's Live Beats show. So subscribe there for clips, shorts, and more goodies at youtube.com slash changelog. And of course, join our totally cool, totally free hacker community Zulip at changelog.com slash community. Have a great weekend.

1:11:06Share changelog with a friend or three who might dig it. And let's talk again real soon.

1:11:31Game on!