#720: Hooked on CloudFormation: GoDaddy stays proactive with AWS CloudFormation Hooks

0:00This is episode 720 of the AWS podcast released on May 12th, 2025. Hello everyone and welcome back to the Adibus Podcast. I'm Alex Sheeran. Great to have you back. I'm joined by not one, but two very special guests today. Firstly, I'm joined by Stella He, who is a Senior Product Manager at AWS. Hi, how are you doing? Good, good to have you here today. And I'm joined all the way from Hawaii, because why wouldn't you mention that if someone's in Hawaii? James Kelly, who's Senior Software Engineer at GoDaddy. G'day James, how are you doing? Hey, happy to be here. Good to have you here. Now we're talking about a topic that is close to the heart of many listeners and should be close to the heart of anyone doing infrastructure as code, which is about cloud formation.

0:44But in particular, we're going to be talking about cloud formation hooks. And so before we get into the whys and wherefores, and James is going to give us some really interesting insight into actually using this, let's talk to someone who helped construct this whole concept and figure out what it should do and why. We're going to talk with Stella. Stella, tell me firstly, let's just do the quick what is CloudFormation for anyone who's not familiar with it and why you would use it. Sure. So AWS CloudFormation is a service that helps you define and manage your cloud infrastructure using code. So you create templates, you can write it in JSON or VML that describe the resources you need, like servers, databases, or networks, that CloudFormation automatically provisions and configures these resources for you.

1:31It makes it so much easier for you to deploy and manage applications in the cloud, especially if you have a complex infrastructure or if you need to make sure that all of your infrastructure are consistent, repeatable, and in any kind of environments. Yeah, I'll emphasize that if you are not doing infrastructure as code using some kind of infrastructure as code tooling, you are definitely doing it wrong. Gone are the days of click ops and random scripts and other bits and pieces. It's got to be as code in a repo with version control managed effectively. Otherwise, you're going to lose your mind.

2:07So an important thing. Now, CloudFormation has been a wonderful thing for a long time. Although I have to say, I've never been more happy than when we introduced YAML support instead of JSON, because, man, if I don't have to debug another JSON, I'll just be very, very happy with that. Many hours of my life, I'll never get back. But we've got something new, CloudFormation hooks, and that's what we're talking about today. So what are CloudFormation hooks? So CloudFormation hooks is a feature that allows you to proactively evaluate your infrastructure as code configurations, or CloudFormation templates in this case that we've been talking.

2:42So you can evaluate this prior to provisioning. So you can invoke a custom logic to inspect your resource configurations prior to create, update, or delete CloudFormation stack operations and the new integrations with AWS Cloud Control API operations. So this is interesting because if we think about CloudFormation, you know, you get to sort of have a declarative approach to say, hey, go make this thing for me. And then, you know, CloudFormation, amongst other things, is kind of being like a state machine, driving the deployment, the status, et cetera. Why did you have to create hooks? Why did the team have to build that for our customers?

3:18So normally you use infrastructure as code or cloud formation to model, proficient, or manage your cloud applications in a safe, predictable, and repeatable way because you have large organizations you manage or just want to do best practices. So to do it, you need a guardrails or something that helps you to move fast but secure, minimize the risk without limiting developer's agility, which is something that is pretty hard to do. There are customers who are relying on detective control. So they evaluate their cloud environment after all of the resources are up there. And this is probably like too late for some customers.

4:04It's like it's already up there. The problem already exists. Thanks for telling me. And we also have other customers who are using Golden Template Plus. They author these templates that only this template access in the entire organization, which is limiting for a lot of developers. They cannot gain the advantage of new IDA services that is out there in every single day as fast as they would like to do. So Hooks provides a way for customers to automatically and proactively enforce their policies early in the development without these issues. Yeah, so in classic, I guess, security sense, we often talk about pushing security to the left of the development stream so earlier.

4:49So this is about really trying to catch things before they get deployed versus waiting for them to get deployed and they're reactively doing it. So it's proactive. So help us understand how Hooks works from a using experience perspective. You can start using hooks by authoring policies. So you define your policies as part of authoring a hook, and then you codify your organizational best practices or your security or your compliance rules to be enforced through hooks. After that, you can configure your CloudFormation hooks to send a warning or block the provisioning of the operations when there is any non-compliant resource configurations found.

5:29Okay, so it's kind of like an automated gate, but you get to choose the details. Tell me more about this building thing. What am I making? You can author these hooks using a Lambda function. We just launched a new pre-built Lambda hook that you can just author it as a Lambda function. Alternatively, you can also use CloudFormationGuard domain-specific language, store it as an S3 object, and run it using the pre-built Guard hook. Lastly, as a last resort, if in any case you cannot leverage these pre-built hooks, you can also create your custom hooks and register the hooks to CloudFormation registry.

6:07So you can kind of choose your own adventure. And James, we're going to come to you and figure out which part of the adventure you chose, that's for sure. But beyond kind of, I guess, stopping bad things happening before they happen, what other ideas do you have for how hooks can be used? So we launched a new integration with AWS Cloud Control API. And this helps a lot of customers to standardize their proactive control regardless of their infrastructure as code tools. So Cloud Control APIs is a service that allows developers to manage AWS resources in a consistent and unified way. It provides a standard interface to create, update, delete, or query cloud resources regardless of the service or the product.

6:52So now that hooks is integrated between CloudFormation and CloudControl API, if you have other infrastructure as code tool aside from CloudFormation, you can still have the same standard and evaluate your resource configuration. Nice. That's handy. Now, James, you'll hear from GoDaddy, a company that probably many people know. In fact, I think I mentioned to you last time we spoke that I have my GoDaddy renewal. I will be paying it for all my domains. but let's start with tell us a bit about GoDaddy and and your role there yeah so I am a senior software engineer with our cloud governance team at GoDaddy so at GoDaddy we have hundreds of developers across all sorts of different teams and they're all constantly deploying to the cloud so we have a cloud mandate that means we deploy all of our workloads to the cloud and we want to make sure that everything is secure like security is mission critical for us but you may have someone who is you know a database expert and they may not be as familiar with the nuances of all of the aws vpc products right so when they say oh i can't connect my database i see this thing that says publicly publicly accessible right so we'll have a control and say no you can't do that um and it proactively ensures that like you said we're not detectively reacting to it where it's already out in the wild and we're proactively ensuring that it doesn't get out there in the first place so So that's been really key for us pivoting to hooks.

8:23And the CloudFormation hooks, they have distracted a lot of the complexity of like a custom solution over to the AWS side. So they're unique because they're the only AWS managed proactive solution for custom controls at least. So you can choose your own adventure because I guess there are lots of ways to tackle this. And you guys were tackling it with some custom stuff you've written. And in many ways, it's the classic um can you tell me you're building this so i don't have to build it myself type thing um that you've kind of just said yep we're using this absolutely yeah we were starting down this whole custom solution but due to the way that different things work it used privilege escalation and different proxies of stacks and the whole thing was really brittle so as soon as we heard that hooks was in alpha i think we immediately pivoted to that and uh the the fact that it's aws managed and supported has just absolutely been a killer feature for enterprise security i mean it makes it super easy for it's powerful for us to configure the controls exactly how we need to but it abstracts away all the complexity of having to design and maintain those system ourselves and we're actually hoping to move to a aws guard for the rules and use the guardhook.

9:38So at the end of the day, we're just writing rules and everything else just kind of works. So you can spend more time on the rules and understanding that because you guys have a really interesting approach where you're very focused on not just saying, well, here's the rules for the organization. You're like, no, well, we can tweak based upon the stacks and the resources and the approach. Tell us more about that because I know a lot of security folks, a lot of developers are struggling with, well, you give me sort of this one size fits all approach and it doesn't work for us. I don't like it. So how do you tackle governance to allow, I guess, that responsible flexibility?

10:12Yeah, that's a great question. One of the very smart design decisions that was made, I can't take credit because it's before I got to the project, but we basically decided that at the account level of granularity, we would allow for any control that we have to have an override and that would be built into the system. and that's really given us the flexibility that we need for those special cases maybe i have you know an account over here where they need this one specific thing but we don't want to enable and open the door for everybody right everyone yeah and in guard we were able to adapt that into the guard hook originally we had kind of a more of a custom lambda hook where we did a lot this custom logic ourselves but we were actually able to do that in the guard dsl as well by setting different input parameters and putting logic in each of the rules that says, hey, if this account has input parameters, which we as the governance team control as the inputs to the hook, that means we're not going to enforce this specific rule.

11:11So we get very granular controls, but we can still have full flexibility. And we can move to that. That's been the big boon for us is the flexibility for the devs. I see that as a huge thing. I want to keep, I guess, pressing into that because, you know, often the governance teams and security teams are seen as the department of no. And, you know, they're somewhat unfairly because also, you know, they have to clean up the mess if a mess is made. But there's often, I think, a cultural aspect too that I'm interested to understand more about because for a lot of folks, they're so beaten down by bad things happening or getting blamed for bad things.

11:48Like, well, that's it. We're just locking it out for everyone. I don't care if the developers complain, tough. And clearly, there's been a conscious conceptual decision here to say we we we recognize the need for flexibility and we'll allow within certain parameters how do you set those parameters like what's the what's the meta of all of this to help folks who maybe are trying to build that into their own organization yeah well you've hit on something that i think is important which is you sacrifice developer velocity in a lot of these more rigid systems so we had a service catalog approach which is another good tool but the way that godaddy had implemented it was not great it was kind of similar to what stella was saying where there's kind of one template and for say an s3 bucket and if you need something different then that's too bad for you um and it didn't allow developers to try new services or different approaches and it really slowed down the development process so at godaddy we highly value experimentation but we also highly value security so um what we ended up doing uh For Hooks is we decided on essentially specific rule categories, and those have to do with other GoDaddy-isms around the AWS cloud.

13:01We have a very specific setup for VPCs and other things. And we know that if essentially you adhere to these categories of rules, that you can have a secure deployment. So we actually just finished a project that I led where we used the CloudFormation reference, the AWS online documentation. we use llm inference to parse through because there's hundreds of these different properties we used to do that manually so that sped that up a lot and that really gives a good starting point for manual review into okay for a new service that maybe i'm not familiar with but a developer wants to leverage what are the potential you know is there things that need to be encrypted in transit or encrypted at rest are there things that could be publicly accessible over a network having those uh declared for your your org if they're kind of special for your setup really helps you find the controls you need to have i think it's really fascinating how do you i guess manage that stream of of requests of hey i want to do something a little bit different or unusual like do you have a structured process is it on demand help us understand that yeah it is uh kind of a structured process where we try to get some head start from a team you know before they want to go and use something and like I said we went in it's just kind of a very basic kind of I guess being called agentic AI now but it's a you know we're just have a little bit of structure around the prompts where we'll feed in all of the CloudFormation documentation which is an awesome natural language source of documentation for every CloudFormation resource type and every property of every resource type.

14:44Just this huge amount of information, feed that in. And that gets a good starting point, I'd say like an 80 % for us to jump off and go in, we engage with our security team, we engage with the team who's implementing it. And we engage with our AWS technical account managers to really understand what the control should be. We finalize those. and once we have that in place we can turn it on not just for that one team but for all the teams at GoDaddy in a secure fashion. It's interesting to hear again I think one of the benefits of LLMs etc is that just that processing of reams of code and reams of information that can be impenetrable I know from a security professional standpoint you feel almost overwhelmed to try and keep across every single thing and now you kind of can if you know how to prompt it the right way you're getting the right answers for yourself.

15:32Yeah it's definitely getting there. And we've seen, in our case, I mean, there's over, there's over 240 different CloudFormation services and growing every day, which is cool. But there's also times when a team comes and they say, Hey, I want this new service. I've never heard of this thing before. Are you making that up? Is it April 1? Yeah. Well, it's every, you know, every new day, Amazon's coming out with new stuff, which is, it's powerful, but it's also hard to keep up with, like you said. So for us, the key has been getting us to that 80 % or 90 % as a jumping off point, sure, manual effort and review versus before where we were was doing everything from scratch.

16:12And I can tell you that there are 167 properties of the AWS S3 bucket resource type. I have read all of them. You'll now name them. Yeah, I would recommend that as an approach. And you can still have very high level of security, especially if you have you know again custom controls for things that your enterprise cares about uh above and beyond the general ones for sure now you talked about i guess a behavioral change or an environmental change when you sort of were able to move away from a classic service catalog approach to one where i guess you know any cloud formation based tool is going to work in this environment let's let's unpick that a little bit because it's interesting how and for a lot of folks, they may never use CloudFormation, but they use the SAM framework or they may use CDK.

17:04So help us understand how that's affected the developer experience. Yeah. I mean, with an organization as big as we are, I think we have at least 500 developers that I've seen in a single, you know, like Zoom meeting. And having all those people, you know, you have different approaches, different tools that people want to use, all sorts of stuff. And in service catalog, it was very much for our setup. Again, the way that we had implemented it, you had to do things in a very specific way. And the whole reason for us to undertake this effort was to increase developer velocity. So now they have the freedom, not only to use whatever services they want, and whatever resource types they want, they can also use whatever tooling they want.

17:49So we recommend AWS CDK, because it is AWS supported. We also think it's a very good tool but they're also free to use other tools and we're bringing on cloud control api support like stella mentioned which enables the use of other tools including things like terraform that's a real cloud crowd pleaser and some other things with the same rule set so we're not writing any more rules we're writing them once and we're giving the flexibility to our developers to say okay you can use any sort of you can use sam you can use whatever tooling you want we're going to evaluate it with the same rules and it's up to you to choose your own adventure there that's very cool because i guess again from a from a classic security mentality perspective it's like i want one way to do things i want one set of rules you know if every time you add a new thing it's now um you know exponentially more complicated for me etc and this is trying to overcome that while still providing i guess you know that great developer ergonomics because if there's one thing i've seen that affects developer productivity it's the ergonomics of the developer environment and there is no one developer environment everyone has their own preferences and everyone's right right yeah and you know especially with all of the new opportunities opened up by llms and things of that nature it's more important than ever for us to be able to validate ideas quickly and enable you know teams to chase after something if they see value there and if there is you know iterate on that and refine it and it's so much faster now that i mean you used to have to for again our system had to open a pull request against this template wait for someone to merge it i think it'll get deployed tomorrow then you can see if you made the right change so just leaks faster that's so cool that's so cool now you've you've you and the team have jumped onto using hooks really early on um so much so you have the t-shirt You have the Hooked on Hooks t-shirt, limited edition t-shirt.

19:46I don't have one of those, but you've got one. So you deserve it because you've done the work. What suggestions would you make for listeners who are thinking about exploring hooks or maybe you've started using it because you've clearly learned a lot. What are some of the deep down tips you'd give us? Yeah. So we adopted hooks before a lot of these cool new features that were announced at the last re-invent where I got this nice t-shirt from the team. and those are where I would start. So the AWS console, actually, the CloudFormation team completely redid it. There's an awesome GUI section that has documentation and stuff for people just getting started.

20:27I'd recommend starting with the pre-built guard hook. It's completely free to use, which is a nice perk and pairing it with a managed rule set from the AWS Guard Rule Registry. So it's a predefined rule repository in CloudFormation Guard, and it has a lot of common AWS services to get you started. So you don't have to do any work. You can pull the pre-built hook, you can pull some pre-built rules and get going immediately with those. If you want to start defining your own custom rule set, like I said, the CloudFormation resources property reference, it's excellent source of structured natural language documentation.

21:03And that's a good place to start on your journey if you want to enable some more fringe services that aren't in the registry. That's amazing. And so for those customers who want to go really deep, they want API level control, how detailed can you get? What do you found works? Well, so the other really nice pre-built hook that Stell mentioned is the pre-built Lambda hook. So it used to be that you had to define your own Lambda and register in CloudFormation, which you can still do. There's a lot of extra overhead and little details you have to do. And what they've done is they've made it just as easy as authoring a regular Lambda function.

21:42We're going to give you this event payload. You're going to give us this response and it's just going to work. So if you want to do something crazy, like make a real-time API call, you know, you want to go check the state of the world in some case and make a decision based on that, you absolutely could do that. And it just is a lot simpler with the framework that they've set up now at the last reInvent with these new releases. You can just give the R of this Lambda and then we'll spin it up for you. That's cool. That's nice. Now, with anything people want to get around things, no matter how well-intentioned they are, if I'm trying to do something that is not in cloud formation, can I just bypass it?

22:24Can I get around it? Yeah. Yeah, so a couple tips from how we set up our IAM roles and permissions to kind of pair with this that I would recommend for anybody. So remember, hooks only apply to operations within CloudFormation or CloudControl API. So that means if I'm making a direct API call, something like S3 Create Bucket, it's not going to be governed by hooks. And hooks themselves are also deployed per region. They're not per account. so it's very important to make sure that your users can't accidentally or not bypass the governance by either making a direct api call to create or update a resource or by deploying something in a region that doesn't have a hook provisioned yet so the way we do that is we use the iam aws called via first condition key and if you give that a value of cloudformation.amazon.aws.com that basically says that the api call must originate from cloud formation or cloud control api and then you can pair that with the aws requested region imt and use a array value of regions where you have hooks configured those two conditions together ensure that the user is going through the hook governance and a region that has a hook enabled so you'll want to pair that with your hook deployment once you have that set up you can basically be ensured that for the resource types that you're governing they're going through the hook governance process it's fantastic really really interesting.

23:51Stella, do you see this as an example of how most customers are using hooks? Are you seeing variations? What's been the adoption? I feel like a lot of customers, they are monitoring their detective controls to start and then they look at what are the common issues they have been facing and equivalent of proactive controls. That's usually how they will just set it up and then they implement it across their organization and they will put this in the warn mode where they will still allow the resources to be proficient after a while they will like a couple weeks they will flip the hooks into fail mode where this will block their deployments and then they usually see a decline in their detective controls alarms so it's kind of a soft launch rather than a hard launch it's like we'll just monitor things see behaviors and also i mean far be it for me to say that a security person may get something wrong but we may have a rule that's a little too restrictive that we might want to also adopt.

24:49So that gives us the chance to do that. So it's kind of like feel your way before just bringing down the hammer and saying, you will not deploy this way. Yes. Yes, this is helpful. Who want to test it out and see what's the impact for the organization and move forward to be more governing after that. Nice. So I think that the takeaways here are if you're not using CloudFormation in some form, even if it's not directly, as I mentioned, could be via some other tool, you should be. and if you're using it, then you should be using hooks. So James, thanks so much for coming on the show and sharing with us your own journey and the journey of GoDaddy and the team.

25:23Yeah, thank you for having me. And Stella, thanks so much for unpacking what the team is working. I'm sure they've got lots more in the cupboard coming out as well. It doesn't stop here. There's more coming. Thank you. Fantastic. And we do thank you all for listening and we do love to get your feedback. AWS podcast at amazon.com is the place to do it. And until next time, keep on building.