#741: Modernizing Edge Infrastructure: Booking.com's Journey with AWS CloudFront and Lambda@Edge

29:37So that is awesome that you are using chaos engineering. I hear customers who want to be able to use it. They're a bit scared of being able to use it. to perform chaos engineering? And maybe I should even first ask if you can just explain to the listeners, for anyone who's new to the term, what is chaos engineering? Yeah, sure. So in general, let me start. I always like to start with the why. The only way for you to be able to absorb failure reliably is to fail all the time. And that's exactly what chaos engineering does. It teaches everyone, every service owner, every platform owner, every network owner as well, that we should build our systems to be very fault tolerant.

30:22If any part of our systems is down, we should have ways to automatically auto heal and auto recover. Usually we want that to happen very quickly. And this is what chaos engineering does. We look at all the layers and we double check that every layer is redundant by making part of it fail intentionally. So when it's failing all the time, we are learning all the time, and we are enhancing the reliability all the time. And that's what chaos engineering is in very short terms. Love that explanation. So how often do you just test overall and then implement chaos engineering? We have various policies on various services.

31:04The minimum requirement is to fail over once a quarter. And in some services, we do it almost weekly and even random timings. but in general the minimum requirement we have at booking that every single component at booking especially the high criticality ones have to be failed over at least once a quarter wow so you said fail over once a once a quarter so when you say fail over do you mean like like you're testing once a quarter a quarter of what would happen if you had to fail over yeah it could go it could be anything we don't even announce what is that we do it kind of abruptly so my team for For example, we own DNS infrastructure.

31:47So once a quarter, we go into one of our data centers, the internal DNS, and we just shut off all the internal DNS servers. We stop broadcasting IPs, and all the servers have to now reroute their DNS into a second data center. For example, my load balancer, the one facing the internet, we talked about that received the traffic from CloudFront. Also randomly, without notifying anyone, once a quarter, we would go and shut off all of them in one of the data centers randomly. And everyone in the company, all the architecture around it, next to it, on top of it, have to adapt. And we have a combination of DNS, healthy checks, and any cash, and all the layers that allow us to recover almost instantly without any disruption to the business or very minimal one.

32:32There are a lot of companies that don't do this. They want to. They're already at a stage where they have business critical applications so what advice do you have because this is really like a cultural change as well so what do you advice do you have for a business um to be able to start let's say we can start testing more frequently since some maybe maybe they do it once a month or maybe they just pray that nothing bad happens which is i don't recommend right we didn't get to this overnight it took us years of actually different methods until we get to this point if you are a business and you want to start implementing some sort of case engineering I would start with planned drills where we would announce to everyone and we give them months ahead of time that at that date end of October we are going to shut down this specific region and for every single service owner probably network engineer or platform owner please do your drills do your checks and be ready for that in the beginning of course you will always have little failures.

33:39You would pay the cost of failures as well, but this has a great return of investment in the longer term. You start by plan the drill, then you make those drills more often. You start by doing it once a year, then once a half, and then you start doing it quarterly. You start making this as part of your business as usual, where your reliability organization would be comfortable with doing this. Once you do it more, you build more confidence, then you can start applying policies across the company that every medium criticality has to do it once a year, every high criticality has to do it once a quarter, and you slowly, you get to a point where those failover becomes as well business as usual.

34:23At that point, maybe you can start thinking about introducing Chaos Engineering where you can let teams know I'm hooking up your service to this chaos engineering API, and at some random time, it will be failed, and it will recover after this time, and you should be ready for that. But this is not something you can do overnight. It takes a lot of time to build the culture, to build the automation and tooling, and eventually to get to that point. And then this is so fascinating. And I'll have one last question on this because I know a lot of people are probably wanting me to ask these questions on this topic.

34:58So in the chaos engineering when it happens, do you let the other engineers know after the fact, or they've gone to the point where they kind of know if something were to happen and you were able to be resilient that it was just maybe like a chaos engineering test? I'm not sure I want to keep talking about this topic. It's not really my area, but I can... Not area? Okay. All right. So I'll just remark and just kind of ignore it. So we are a user of it. We are really ready for it. So I'm not even sure how to answer this question. It's okay. All right. Yeah, it's just something I thought of randomly, but okay, I'll put a marker to get rid of that.

35:40Okay, that's fine. Okay, so I'll move on to the next question. Okay, Sarah, so I know there are probably some listeners who are hearing like the architecture that Booking.com has with CloudFront, Lambda, Edge, and they're probably thinking, wow, that's like a really advanced architecture. It's fine that I get that Booking.com can do that, but we're still like a small, medium-sized business. We're still a startup. Is this the type of architecture that also could be feasible for my business? So something to keep in mind is that CloudFront and Lambda at the Edge are services they can be adopted from day one that is regardless whether you're a startup a scale-up or an enterprise so what customers at any scale like about those services is they can offer you as Ali actually mentioned this global reach so that unlocks not only a lot of opportunities and a lot of advances when it comes to the engineering part of the organization but also So new markets, reach to new markets if you're an organization or business.

36:55And all of these services allow you to do that without the burden of managing this complex on-premise infrastructure. So indeed, we need to acknowledge the fact that booking operates at a gigantic scale. Like the numbers that Ali provided and the use case. that obviously that does not apply to startup or scale-up and so on. So Ali talked about migrating this complex on-premise edge network infrastructure, like mini-pops and data centers. Startups and scale-ups wouldn't have that edge network on-premise to manage or migrate in the first case. So they typically begin with this infrastructure deployed directly in the cloud and they would generally adopt platforms and not at the edge as their primarily edge solution from the beginning.

37:49Also important to keep in mind is that also it's not only about management and complexity and this legacy infrastructure on-prem, but it's also about the order of the scale, right? So the traffic, the booking as, as well as the usage of the service is bigger. is bigger. So startups and scale-ups could potentially start with much smaller traffic, but this should offer the reassurance that if your startup scale-up grows, as the business grows, those services will be able to handle the growing traffic and the growing usage with high availability and high scalability. And that is important to keep in mind.

38:34So these services are accessible and do fit different use cases, regardless of your company size. That being said, so I just also want to add, give a shout out to Ali, because to be fair, I do feel that even the Booking, and especially Ali's team, Booking is a large enterprise, right? Ali's team really moved with the agility of a startup. So shout out to him and his team, because, I mean, I say this in the best way possible. So everything is a journey, but I think his journey was quite effective and fast. So congratulations to him and the team. Very good, sir. All right. So maybe you can tell me more about this architecture that you've now adopted, CloudFront.

39:22But let's talk more about the Lambda at Edge part and how you're using Lambda at Edge. Yeah, sure. So very early on when we were thinking about CloudFront, this Lambda, the Edge thing, keeps popping up. It seems to be a great opportunity. Typically or historically, we used multiple types of reverse proxies in front of our traffic. We had very limited capability of executing business models. Usually it is something very simple. We use some newer scripts. We use some native configuration in the reverse proxies just to manipulate a header, remove a header, add something, some security stuff, and that's it.

39:59Then once we saw Lambda on the edge, a big opportunity presented ourselves. Now we can execute very sophisticated logic on the edge written with Node.js or with Python, where we can do very sophisticated and complicated logic on the edge. And as mentioned, we have a very complex ecosystem. We have all sorts of infrastructure going from private cloud to public cloud to monolith, etc., etc. We never had one unified layer of business logic on top of everything. So that's very quickly presented an opportunity where we had teams in Booking internally piloting who should own that Lambda on the H-Scane.

40:44We are very limited. We can only execute a few Lambdas per request. We can execute a Lambda on the viewer request to the edge, and we can execute a Lambda on the origin request from the edge to the origin. And then we can execute Lambdas on the way back in the responses from the origin and in the viewer. We had at least 10 use cases who presented to my team that we should be the ones taking that Lambda on the edge team. It's great for us. we will have great benefits for business. All the way from authentication to traceability to security headers, to bot control to local and language and currency, all sorts of amazing things that could contribute to the business quite a lot.

41:35So me and my team, we took a step back and we thought, okay, seems like this Lambda and the Edge thing would be great for multiple people in the team. Let's actually build a platform on top of Lambda on the Edge To allow all those modules that make sense To exist on the Edge side by side What I mean by that We set requirements We need a way to allow all those modules to run Ideally in parallel We need a way to make sure Lambda on the Edge is super critical If there is a failure there, it will fail all our path So we needed a way to have also failsafe there. And if there is any module who have an issue, we don't want it to take our whole website down.

42:19We needed a way to make it configurable and to build some firefighting tools where we can enable, disable a module with a click of a button. We took all those requirements and many more, including the latency where we needed it to be less than. In the beginning, we set a number as 50 milliseconds as a maximum we want to add to every request. and then we designed our own framework. We built it with TypeScript where anyone at Hooking who wants to implement something at Lambda on the Edge, they can just import our interface, implement their hand in function and we have our own tooling who would take all those modules, right now we have I think around 12 of them, and bundle them into one zip pile that we deploy on Lambda on the Edge everywhere.

43:02And our function runner will pseudo try to execute to those modules in parallel and in isolated kind of mimic sandboxes. If a module have an issue, it will just tail that module. If a module is trying to run beyond the set time, we will just time it out and let the request continue. So, and that's where we stand right now. We did a lot of optimization, trying to find the right Lambda size, adding traceability, connecting it to open telemetry, and optimizing on the performance of the different modules. Because now it's global, we needed to find a way to also get data into the edge, whether that could be DynamoDB global tables or using a CloudFront and Assetree next to Lambda at the edge.

43:52It's a long journey with a lot of details. I don't think we have time to dig into all of it. But in the end, we ended up with this amazing Lambda at the edge platform that allowed us to do a lot of interesting stuff. Just to give two examples, I mentioned in the beginning at Booking, it is a religion for us to be data-centric and to run experiments on top of everything. One of the amazing things we added during a recent EBA working very closely with AWS to fast track part of our infrastructure into AWS is to move the experimentation system into the edge where for the first time in booking history, we can run experiments trying out some services between our legacy infrastructure and our modernized infrastructure in AWS.

44:42And all that is happening on Lambda on the Edge, the coin toss, the blob object, and even the addition of cookies and et cetera. Authentication was another one. For the first time in booking history, we have a way to cover all sorts of infrastructure with one authentication cookie that covers everything. And actually, one of the side effects of that, it made our customers, our service owners, free from being tightly coupled to some legacy infrastructure. For example, by moving authentication, boot control, and experimentation system to the edge, now a module owner inside our legacy monolith application is not stuck there anymore.

45:26They can move into a modernized Java, containerized application running on AWS. And because all those good stock is running on the edge, they get it out of the box without needing to get stuck where they are. I'd love to understand maybe what you saw as the business impact of now moving a lot of this logic now at the edge. Well, the very short answer is accelerating modernization. As I mentioned, we had the goal for a long time to modernize part of our infrastructure. And that was usually very hard, mainly because of those common middlewares, very old, heavy, complex logic that lives as a form of middleware inside our legacy systems.

46:16by moving this into the edge and the edge being a cover of all other kind of infrastructure that really freed our service owners to easily move between platforms, moving from bare metal to private cloud to our Kubernetes cluster, even to native AWS solutions, basically using an API gateway within our Lambda. So Sarah, I know earlier you were talking about Lambda at the edge, but it would be really helpful to hear from your perspective about how customers can use Lambda to really get more granular with their own business logic at the edge. Yeah, so what I like about this service personally is that it's an intersection between serverless and edge network.

47:05So it's not about running any code on a serverless environment, but running code specifically at the network edge, which means for you and for your business, closer to your end user. So Ali already gave a great overview of how Booking uses Lambda at the Edge capabilities. So customers can use Lambda at the Edge to intercept those HTTP requests before they hit your original stack on AWS. And also HTTP responses, of course, after they leave that origin, that original stack, And before they reach your viewer, so your clients, your browser or mobile apps and so on. So I've seen many customers use Lambda at the Edge to route requests to different regions, for instance, based on user agent metadata, information about the location, device type.

47:59Booking already leveraged a lot of these capabilities, like A-B testing, experimentation. the capability really unlocks experimentation, which again, it's tied not only about on technical metrics, but also business metrics. So you're able to quantify the effectiveness of something that you have changed in your system or in your platform closer to the edge. And you can also add some totally custom business logic, right? And that customization, so executing code at the edge, also allows you to customize content based on other parameters. But not only that, you can also implement redirects, for instance.

48:48Just as a reminder, that request-response manipulation that the customer can set can be done on a viewer request, on an origin request, on an origin response, and a viewer response. Outside of A-B testing, I've seen customers doing traffic splitting as well. You can personalize your content. And so that locks also a lot of capability when it comes to providing an enhanced experience to your own end users. Ali also mentioned a little bit about securing authentication. I've seen customers using it for validating token or credentials at the edge. So reducing the burden and the load on the regional stack.

49:39Ali also mentioned both protection. That's another very common use case. You can add security headers and that indeed. But also, as we mentioned, enhancing business metrics, A-B testing, experimentation, and so on. You can also use it to do performance optimization. So dynamic compression, image optimization. I did indeed mention at the beginning that some media companies use it to manipulate, to manifest to five video files. That's another cool use case in my perspective. So you can do real-time content transformation. And I think all of this can really unlock a lot of opportunities for businesses.

50:27At the same time, it can really accelerate the speed of innovation that will benefit your own customers as well. Wow, it's really amazing just how many use cases there are for Lambda at the Edge. And it really sounds like a lot of customers that aren't utilizing Lambda at Edge, they already have an architecture where they're just doing this wherever it is that they're doing this. So it requires, I think, a different level of thinking of how to architect your application. So, Sarah, I'm curious from your perspective of any advice or how customers should think about architecting to be able to shift some of this logic that they're already doing, like device type A-B testing, over to the edge.

51:22Yeah, so great question. So I would apply a lot of the best practices that we recommend for serverless computing and Lambda in general. So a lot of this was covered by Ali and also delivered by booking as part of their journey. Ali mentioned something around troubleshooting and observability. So being able to have the dashboard, monitoring dashboard and alerts in praise to be able to troubleshoot when the scale grows and when your business is successful as well, you want to minimize decent impact. So this is something that I would definitely leverage. Another aspect that also Ali mentioned is cost management.

52:07So Lambda at the edge is charged based on two factors, two dimensions. One is the number of requests and the function duration. So when you're a startup, those two dimensions may not be as impactful in your cost eventually. But at scale, small efficiency can contribute to unnecessary cost. So make sure, and this is something that I advise a typical customer to do, to really be mindful about the right trigger that you use to invoke your Lambda at the edge. So do you want your function to be executed, for instance, when there is a cache miss? Then you should use origin triggers, perhaps. And do you want your function to be executed for all requests?

52:57Then maybe you should use a viewer trigger. So think about what kind of events are best suited. So avoid inefficiencies. When it comes to cost, CloudWatch costs should also be taken into account. So login outputs need to be really fine-tuned. Adopt best practices like log sampling and selecting the right log level. Make sure you use structural logging so that you're able to understand and troubleshoot when something happens. But all of this selected in the RILO level, log sampling to a specific percentage of requests, and in production, perhaps only logs, errors instead of the whole request invocation.

53:49Outside of log sampling, of course, that is a good way to keep custom under control. But in general, also other best practices when it comes to Lambda at the edge is performance optimization. So which means keep your Lambda code as lean as small as possible. Minimize dependencies. So if you're using Node.js, you can also use 3Shaking, for instance. That's a typical way to kind of remove all the dependencies that you don't need. So keep your code as lean as possible. So your Lambda not only is going to be more performant, You're going to reduce your cold start and you're going to reduce also your duration, which has a positive impact on the overall cost.

54:33So these are kind of the pillars that I would keep in mind, especially as your company grows. Performance optimization, good monitoring, observability in place, cost management and fine tuning. be mindful about this so you can transfer a lot of the knowledge from the knowledge that you may already have from Lambda, the original stack. But yeah, this is whatever the advice customer typically. Great advice, Sarah. I would just add to that. The performance optimization was a very important one for us. We spent a lot of time trying to lower the size of the total Lambda zip file with the tree shaking and many other techniques.

55:19we managed to get it into the right size and playing, tweaking with the fine tuning of Node.js stuff, our code, and also playing with different sizes of Lambda on the edge. We managed to lower our latency, average latency request from 50 we started with to 40 down to 12. And now it's around only five milliseconds on average for the request. And that was mainly due to finding the right size of Lambda. Too much was too expensive, and it was actually adding latency, and too small was also not great enough because it added a lot to our code stats. Just fine-tuning, playing with it, we found the right edge on the end.

56:08And that's still, I need to remind you, running 10 modules in parallel in that four-millisecond average. Wow, I absolutely just love all of those recommendations recommendations and Ali, just how it is that you've been able to apply them at booking from performance optimization and even cost optimization. So I want to get into that as well, because I love that Sarah was getting into cost optimization as well. This is a topic that is always top of mind for customers. So Ali, I'd love to hear from your experience with cost optimization with this entire architecture. Yeah, so there is a lot to be said about cost optimization.

56:47In addition to what I said about observability cost, about optimizing the right size of Lambda, really iterating multiple times over the efficiency of your code, I would add maybe a story. I think you already mentioned it, where I talked about how we saved half a million with one line of code. Since we started, when you build a normal Lambda, there is a feature that comes with Lambda. When you enable logging, Lambda by default spreads three lines of code, or three lines of logs. Start by when did this Lambda request start, when did it end, and the report. This report will tell you how much resources was consumed by this request and how much total time it consumed.

57:35With the Lambda advanced logging, which is a feature that was enabled only on normal Lambda, but not Lambda on the edge, you can easily, with a click of a button, be saved. Usually these three log lines, they are sent as an API call to CloudWatch. And the average cost, it's different per region, but I think it's around half a dollar per gigabyte. That's the standard cost across AWS. us. When you think about how many trillions and billions of requests we handle, those three lines alone, which is an average of 300 kilobyte, if I'm not mistaken, they add up to tens of thousands of gigabytes, which really added up into a huge sum that we see in CloudFront.

58:20So for a while, we tried to find a way to delete those or disable logging, but we couldn't do that. So we had to engage very closely with AWS. With the help of Sarah and other solution architects, we escalated that and we said, yeah, we need the same advanced controls and logs on Lambda on the Edge like we have a normal Lambda, because this is costing us a lot of money without any real business value. So yeah, eventually AWS engaged with us. They took a requirement and they took action by enabling the advanced logging feature also on Lambda on the Edge. And yeah, it literally took us one little line in our Terraform model to disable those standard logs, things that we don't use.

59:01And it really saved us 10 ,000 months, which add up to more than half a million a year. This is absolutely amazing. I know I've learned a lot. I know the listeners here have learned a lot. So one last question for each of you. Ali, is there any other piece of advice that you would have for customers who are thinking about moving to more of like an edge-based type of architecture? Piece of advice for people using edge. I would say don't be afraid to engage with AWS and ask the hard questions. Multiple times I noticed my engineers treating AWS like a black box and not really wanting to engage deeply with AWS and ask the hard question of trying to uncover what happens under the hood.

59:56But what I discovered working very closely with solution architects, product managers, and also support engineers, that they are very receptive to feedback and they are very engaged, often with a customer-centric view. When we bring a problem or an issue we're trying to solve, we usually get feedback more than we expect, especially when we are really trying to optimize on the small, finite tuning things. I have a lot of great solution architects in mind who were super helpful to us, from people who are network-focused to CDN-focused, and also the product managers who would engage with us and take our requirements way ahead of the market release of some features for under NDA, where they would exactly get the features that we need to do something.

1:00:49Like maybe a great example that Sarah was engaged on, this recent SaaS feature that CloudFrontor published. We had a requirement of onboarding thousands of domains to CloudFront, but we did not need to handle SSL certificates issuing and create an independent CloudFront distribution for each one of those. So we've been engaged with AWS for almost a year now. They were collecting the requirements from us to be an early adapter of this new feature that will allow us to disconnect the front end from the distribution of the backend, making us use CloudFront more like a reverse proxy. And yeah, this is my advice.

1:01:29So just don't be afraid to nag over AWS and ask for exactly what you need. They might say no, they might say yes, But if they say yes, you would get exactly what you want. Really good. Sarah, what about you? So I just want to, first of all, thank you. Thank you for the shout out, Ali. That really means a lot. But for me, I had twofold advice. So I want to echo what Ali just said. Don't underestimate the value and the influence that you can have as a customer over the roadmap of a service, in this case, Edge Services. So please, please engage with service teams and make your solutions architect working with your account team create these feature requests on your behalf.

1:02:22So make them be your advocate within AWS so we can kind of improve our platform and tooling and capabilities. So that is point number one. And point number two is, at the end, this tech, technology, these features, these capabilities, these are means to a goal. So as an engineer, if you're part of your engineering organization, knowing the flexibility and the broad spectrum of capabilities, and even listening for how Ali and Booking is using Lambda at the Edge and CloudFront, think about which markets you can unlock that you haven't unlocked as a business. So partner with your business stakeholders within your company to really leverage those capabilities to unlock new business opportunities.

1:03:14And it's not only about speed of innovation, but it's also opportunities that can enhance your own customer experience and make you evolve your own product. So think about that as well. Love it. Really good piece of advice. This has been such a fascinating conversation. Ali, Sarah, thank you so much for being here on the AWS podcast Thank you for having us Yeah, thank you, Julian And thank you, Sarah, for inviting me as well