OpenTofu with Cory O’Daniel and Malcolm Matalka

22:55But if you have an idea and you say, hey, I did the pull request and someone review it, that'll also get it in there fast as well. Okay. And then can you tell me a little bit about stateful encryption? And is that a feature, am I correct, in it being a deviation from Terraform? Was that the first feature that went out that was the net new one? I know there's a lot of them today. Yeah, that came in like the premier release, right? Yeah. And that one's important too, right? Like if you're using any of our tools, right? Like A, it's important to secure, right? Like it might have a password or whatever, but honestly, like if you're using any, like if you're not running, you know, an open source tool yourself for managing your state and your Terraform or OpenTofu runs, and you're using something like M0, Harness, MassDriver, Spacelift, TerraTeam, TerraMate, anybody, you don't want us to know your usage details of the cloud, right?

23:49Like people are always worried about the passwordness, but like the ability to sell the usage data and the information behind the scenes, like encrypting that is more than just encrypting some secrets. Like it's a really important one. That was one that sat stagnant and open for years. And there was excuse after excuse as to why it couldn't come to be. And it's like, well, it was pretty easy. I mean, for eight companies that never contributed to anything, like we got it out pretty quickly and it's pretty resilient, right? And so it's like, it wasn't as hard as it was made out to be or impossible to integrate into systems that are managing state like it was made out to be.

24:22Just do you want that data visible or not? I feel like was like the real question. And somebody was like, yes, we want to be able to see it. Right? So that was one of the first ones. It's an awesome feature. It's very easy to use. It's driven by a variable so you can pass in that key for encryption. There's a lot of different mechanisms for decrypting and rotating keys and whatnot. So I'd say if you're flipping to OpenTofu today, it's easy to just like throw in a GitHub action for a plan and like see that it's all working. Now, the one thing, the one little bit of lock-in that you'll get is as soon as you bite on that security.

24:55If you want to go back to Terraform, guess what? Your state's in the air. In terms of the encryption key management, is that done through like a cloud KMS? Yeah, there's like a whole, geez, I can't remember exactly how many drivers there are behind the scenes, but there are a whole host of them. Yeah, there's three major ones right now. One of them is just you present the key to open Tofu, however you got it on your own, and it does it there. And then there's GCP and AWS ones as well. Okay, so I can bring sort of my own KMS if I want, but I can also do your line. And also you can go back and forth between an encrypted state and an unencrypted state.

25:34So if you decide, you know, the key management is kind of not what you want right now, or you're just not, it's not important enough to you, you can always go back after you've played with it a little while. That's one thing that's really great about the OpenTofu mentality is there's also, we have migration docs and then switching out is easy enough as well. So there's never this point where you switch over to OpenTofu and then you're stuck with it. You mentioned this earlier, Corey, the OpenTofu registry and how that was one of the first big challenges of the fork. Can you talk about the process to essentially build the registry, how you went about discovering versioning modules and have those providers within the registry?

26:11Yeah. So, I mean, the module spec is pretty well defined. Like you have to go around the Terraform site a bit to like figure out how it works. So under the hood, it's mostly just redirects. So like mostly of what you see in the Terraform registry is just mappings back to like a Git repo. Right. And so we just kind of had to like go through Terraform, figure out how that worked. But then it was like, okay, well, how are we going to host this thing? Like who's going to run it? Right. And that was kind of another concern is like we didn't want necessarily like, oh, you know, mass driver will dedicate the infrastructure and own the registry.

26:40I mean, I would have loved that as a business owner, right? It's a great little thing to get on the bottom of the site and a bunch of information to get. But at the same time, like we don't want, again, any company having information about like the download rates of these modules and providers, right? So it was important that A, that registry was open source so people can see how it works and B, that we get it hosted by somebody besides us. And so I think that was like, that may have been like our big first partnership. I'm throwing that in there quotes with, I don't want to say the wrong name, Cloudflare, right?

27:09Yeah, sorry. I mix up them and one other company at Fastly all the time. But I think that was the first big partnership, right? And it's funny to look at it, but the amount of thought and passion for the community and how they will perceive it goes into this. And I think this is one of our big problems is we haven't done a great job of talking about that passion that we have for building that trust, right? It would have been extremely easy for any of us to have made just a registry and thrown some compute at it. I've got more AWS credits than I can deal with. MassDriver could have thrown credits at it.

27:41But the importance of that being in the public eye and being owned by the foundation and hosted by somebody that's not one of us was also key. Like that is a part of the decision making process. And I think that's one of the things that builds this trust, even though we don't, you know, make a stink about it. But I honestly think we, again, like, I think that's where we haven't done a great job is like talking about like thinking through that process and making sure that it works for everyone outside of the organizations that are involved. Yeah, definitely. I mean, independence is so important for OpenTofu.

28:11And I don't think we do as good of a job as we should to show it's really, yes, there's money coming from various companies to pay for these employees, but it's really independent in every other way. In terms of, you know, populating and keeping the registry up to date, did you run into any like rate limit challenges with the GitHub APIs? I don't think so. The registry isn't, it's not like something that is updated super fast or like, like it's not like Google indexing the web all the time. It's really more of a batch process. And the, the GitHub rate limits are actually pretty high as long as you identify yourself.

28:47You're continuing to use HashiCorp's configuration language. Like, are there considerations or future plans around enhancing or evolving HCL specifically for OpenDufu? I'm not positive. I think that would be, I mean, honestly, like somebody on the Open Tofu team submitting PRs to HCL, I could see that like if we need something or if we find a bug. And I would honestly hope that in the idea of community that they would accept that. But I mean, I don't think that we're going to diverge from HCL. That's maintained its license. I think, you know, changing that is a very nuclear option. So I think HCL and the providers are kind of both fall under that umbrella of like, if those licenses are changed, like the ripple effects there are catastrophic.

29:34So I think we can just depend on that for the foreseeable future. And I mean, if we have to make changes to HCL, we'll obviously submit PRs to it. And hopefully that goes through. Yeah. And I think if you look at the, so if you go to the OpenTofu on GitHub, you can see the project, you can see what's planned for the various releases. They're usually like one or two out. And if you look at what's limiting in the existing Terraform world and now Tofu, it's really not language specific. It's functionality. So a big one coming out now is dynamic providers, which let you say, hey, I want AWS across all of these different regions.

30:13I don't want to make a new provider for each one. I just want to like reference it like you would a dictionary in Python or something like that. And all of that fits well within HCL. It's really about the semantics of how you interpret that HCL and then how you turn that into an actual plan and all that. So really, I think most people are content with what HCL can do and just getting more functionality in there. In terms of some of the deviations that you've made from the Terraform, we talked about the stateful encryptions, but what are some of the other changes that you've made? Yeah, I think one of the things, so there's a bunch, I can go through a couple of them.

30:49But one of the things I think is that I want to point out first before talking about the rest of them is another one of these ones that's important, right? So as Malcolm mentioned, not only are there guides for how to get into OpenTofu, but how to get out if it's not the right thing for you. And so one of the things we introduced fairly early on is the.tofu extension, with the idea being that Terraform will only touch a.tf file. And so if you want to start using OpenTofu specific functionality, you can put it in a.tofu file. OpenTofu reads all the files, right? And so you can actually have your plans running side by side and Terraform will only see the Terraform stuff.

31:22Tofu will see the entire thing. Right. But that's also important for adoption. It's important for like easing that onboarding story of an ops team that's just overwhelmed with a million things to do. Like I can start trying out Open Tofu without just absolutely blowing up all my pipelines and having to do all this painful stuff. Right. So it's just like another place where it's like we don't want to make this hard for anybody to adopt. because we know that these teams already have enough stuff on their plate. But the stateful encryption, obviously super cool. We talked about provider-defined functions, but one of the other things I think is so rad, I'm a big, big, big, big, big fan of using Terraform, OpenSofu now, obviously, as building operational abstractions for my engineering partners.

32:07When you look at the cloud, every API is like half ops, half dev. right? Like I don't like teams having to conflate these things in their heads. And so I like building really abstractful modules. And so one of the things that landed was we have a Go and a Lua provider where you can actually just toss in a main.go or a Lua file, and you can actually write Go that will now execute as a part of your Open Tofu module executing, which allows you to start doing some really interesting expressions, right? So like build out logic. So you can actually codify like your processes in, right? And so I'm already seeing things like we're like, okay, based off of the types of data and like the team, it'll hit an API and pull that team's cost center, right?

32:52Like something dumb, something silly, but like who remembers their cost center ID and remembers to enter it, right? And like being able to kind of hide that away from engineers where it's just like, you just get what you want and like your cost center stuff's going to be in there. We don't have to worry about like associating costs for like, you know, bigger projects. It's done for you. you kind of being able to get to do that because you have this like richer programming language that can sit behind the scenes, but still in this declarative model. So that was one that I, like when I saw it, I was like, oh my gosh, like I've been, like, if you see my Terraform code, I just have local blocks.

33:23They're just huge. There's just loops and maps and all this logic that I could try to build into locals. And now I'm just like, oh, I can just write, I can just write some Go for my complex expressions. So that was one that I was stoked about. Malcolm Go, I don't know what Malcolm's favorite one is, but like that was, I was dancing the day that landed. So my favorite is actually ridiculously mundane, but I like it because I think it opens up a lot more automated functionality. And that is early variable evaluation. And one of the things that lets you do is say you're referencing a module. You used to have to hard code the entire path to the module in your HCL.

34:00But now you can make that a variable, possibly environment variable or environment file, which has a very simple syntax and reference that variable in the module source path. So what's that let you do? Well, now you can do very easily more depend about type things where it's really easy to automatically update these versions now. So if your organization chain updates a module, you can feed it through your entire organization automatically with very little logic and like very little smarts associated with it. And so I really like that one. And it's one of those ones too, that was just on the HashiCorp issue tracker forever.

34:35I was like, no, no, no, no, no, no, no, no, no. And then OpenTofu was like, oh, it's done. Here it is. Have fun. Yeah. I mean, a lot of these things, like they're fairly, like they sound on the surface, like fairly simple, but they make people's lives so much better. You know, you're, you're removing sort of that death by a thousand cuts situations that you tend to run into with, you know, some of this configuration. In terms of like using infrastructure as code, you know, OpenTofu or otherwise, like how do you go about, you know, teams typically go about debugging and error handling like infrastructure changes?

35:09It's hard. There's not great tooling for it across the board. Right. And so, you know, many teams, they have a staging version and they're prod and they'll just like deploy it and like, but like the thing that sucks about all infrastructure is none of your configuration matters. if the apps that are running on top of it don't work, right? And so you can stand something up and be like, oh, OpenTofu, apply was green. I can see the thing in the cloud and it's healthy. But like, until you put an app on it and then verify like the functionality that you're trying to get to happen, like none of that matters, right?

35:42And so there's like levels to testing. It's like, hey, is the code, you know, validate? Is the code valid, right? Does it apply, right? A lot of people that are newer to infrastructure and cloud sometimes think, oh, it applied. Everything is good. But it's like, that doesn't mean you didn't have an outage for 45 minutes while it was applying. Right. And so like that stuff is just very hard to catch. And so, you know, patterns that we do is we like internally master, like how we do our own is we define use cases for each piece of infrastructure, like what it's going to do. And we use, there's Terraform and OpenTofu test now that'll like test your code, but the Terragrunt team made TerraTest, right?

36:20And it's an actual, you write all your tests and go, but you can stand up other things besides your infrastructure. So you can run it. It'll do your Terraform apply. And so we'll do things like, hey, we made this queue and we're actually going to push data into it with TerraTest and then pull data off of it just to make sure that like, is the IAM right? Is the IAM right? You make a queue all day long, right? But if the IAM isn't what you expect it to be, that app's going to fail when it goes in the prod, right? So we do a lot of rigorous testing from the developer view of that infrastructure world.

36:54It's more rigorous, but we don't have a lot of outages. And we kind of build libraries for all the different things that we want to do. So it makes it real nice and reusable. I think that's one of the best ways to think about it. I mean, the number one cause of outages today is configuration changes. So maybe we should be thinking a little bit higher than just like, did it apply green? Did it exit zero? And how does it function from the intended use case? And we use those use cases as drivers for our test suite. So we'll say like, hey, like, you know, we support Postgres in three versions internally.

37:26We have, you know, a development version, which is like a single zone, you know, fairly straightforward. We have development serverless if we're spinning up something for like a preview environment. And then we have like a production config that does like our multi-zone and stuff like that. And our test for our production one will nuke a zone. Like it'll nuke one of the instances in one of the zones to make sure that we're still able to read and write. Right. And like that right there, like the developer view of the world, like the operational day two of the world is the test that I want to assert, not simply that did it make a Postgres?

37:57Like that doesn't matter. Like, can I use a Postgres? Does it have the resiliency that as an ops person I'm trying to guarantee? And so you have to reach outside of pretty much any IAC tool to do that. Yeah, I think you really have to appreciate one thing about infrastructure as well, is that it corresponds to a physical thing in the world. One of those physical things can only exist at one point in time. And I know it's very convenient for us to think about infrastructure as code like software, but software, we generate an artifact and we can run tests against it. And if those tests pass, we can deploy it to our infrastructure and it'll act the same in those two cases.

38:37But dev infrastructure will never be and can never be the same thing as your prod infrastructure. So even if everything works great in dev, you don't really know if it works in prod. So if you are doing tests, just be aware of that and understand what you're going to get out of doing tests in dev and what places they will or won't apply to in prod. So it's just a slightly different mentality in how all of that works. Yeah. I think a really good, like I'll spare everyone the details, but like a really good example of what he just said is I have a buddy who works for a big public company and they have hit the IP limits of Kubernetes in AWS.

39:18You can't test that, right? Like that is a very hard thing to test in an environment, right? So like the reality is like, eh, when you get to prod, like prod's prod baby. Like it's prod's going to do it, prod's going to do it. Like, so we can do it like as good as possible to test it. but like you're never going to get prod. Prod is the one and only source of truth. No matter what anybody tells you, like prod is the source of truth. Like what's happening there is happening there. Like to get an exact replica of what you have requires you standing up all the services and getting the traffic, right?

39:47Like you might have something fail because of the amount of ingress that you have. That stuff's kind of hard to test. Yeah. Well, given that, you know, the number one reason for outage is configuration issues. And I think like something like 70 % of data breaches or leakage of information is due to some sort of misconfiguration that led to overprivileged access. And I think, Corey, you mentioned this earlier that most organizations still today are not using these infrastructure as code approaches. Why is that? What is the blocker to adoption of this approach, given that it can help reduce the risk of some of these challenges for organizations?

40:25I want to make sure Malcolm has time to answer this because I think that we have, I think we have very different user bases. I think we both kind of very interesting opinions here. So, you know, like MassDriver, like kind of where we sit in this space, like we have a lot of people that use the platform that like bring their own IAC, but a whole bulk of our customers do not have infrastructure as code. So I actually get to talk to these folks that are like starting to make that decision. And they're like, hey, which IAC tool should I use? Like when I start using MassDriver, the biggest thing is time and time again is how, little as an industry, we invest in our operations teams and our DevOps teams and our platform teams, I think are getting a bit more love.

41:06But like when we don't invest in those teams, like they are, any ops person on that, like that's listening to this, like you are the force multiplier in your business. It's just like waiting to happen. And when you invest in those teams, like that's where you get that 10X engineer, right? Like if you think about software and infrastructure as, a manufacturing line and the products at the end, like ops and DevOps is at the beginning. And if that's shitty and not working well, the whole thing's wrecked. And if that's performance tuned, the world is amazing. But when we don't invest in these teams, they don't have the time to start looking at tools.

41:42They're literally just dealing with fires all day, right? Like I look at DevOps, I know Malcolm's there too, because I see him on like the same threads, but like our DevOps, our Terraform, like our Kubernetes, is like you see these people like that are like discussing like what's going on in their business and like it's harrowing to read it's like oh my god like you forget how good you have it sometimes when you're at org with like well-established devops principles but many of these companies just like their ops teams are just underwater they don't have the time and so it's like hey why aren't you guys doing infrastructure as code it's like dude i work 60 hours a week i hate my job like i can't work another 80 hours a week but that's the thing that sucks is if you put in that little bit of effort into the automation and reproducibility, it pays dividends.

42:22So I think it's all about just finding the time. And I think that one thing that we've been brutally bad at as an industry and from the ops to DevOps transition is marketing what we do within a business. I remember one of my first startups, it was a video streaming platform in 2008. Imagine that. 2008, video streaming platform in the browser. And our CEO walked by one day and was like, I don't understand what you do here. And I was like, you know how this thing has never been down? That's what I do. Right. And like, that was the moment that I realized I was like, I work my ass off. Like we had, I remember that very early 2000s, we had 5 ,000 people watching a video stream of the Mythbusters like on our platform and the thing stayed up.

43:08And I was just like, ah, I didn't realize that we were that good. I thought this thing was going to fall apart. And this person that was literally two seats away from me was like, I don't know what you do. And I never talked about it. And I think that on the teams, we are these people that maybe might be a little less social. We're definitely less boastful, especially if we're a front-end engineer. But sorry, guys. I love you guys too. But I think we don't do enough to show the importance and the impact that we have in businesses. And that's why a lot of these teams kind of get loaded with all this debt.

43:37They get loaded with so much like death by a thousand cuts all day long. They just don't get the time to embrace that original idea of DevOps, that Kaizen, that continuous improvement. And if you don't have the time to do it, you just don't have the time to do it. But that's where you got to find it. It's like you got to find that small project, that little thing that has a big impact in your business, whatever it is, that's going to take some of the stuff off of your plate so you have the time to start investing in it. And that's what we see almost every single time that somebody comes to a mass driver and they're like, we haven't picked an IAC tool yet.

44:07We haven't started doing IAC. Like, how do we even get started? I think the other thing for them that is also a part of this like death by a thousand cuts is, okay, we're going to adopt IAC. Well, guess what? The business has been around for eight years and we have about 40 ,000 things in the cloud. So how do we even like go backwards? Right? Like that itself is harrowing, right? And I think the right answer is don't just pause on that. Like, because that is a huge endeavor and it is painful. What do you have coming up next that you can start to use IAC for that's net new? Great. Like now you can reproduce it, right?

44:41Now you can understand it, right? That's going to buy you time, right? Let's do that again. And then when we get to the point where we got a really good established baseline for how our business works around infrastructure as code, now let's go look at the big bag of stuff that we got to bring in. But looking at everything you have today and saying, that's how we're going to start, that's a death knell. Like you're doomed. Yeah. Yeah, I mean, I think Ops probably needs to hire a new PR firm. But I mean, the big challenge, I think, of a lot of the back-end work that happens is it's always easier for organizations to justify clearly customer-facing applications, right?

45:17If it's front-end, I don't need to understand the engineering work that goes into it to see that, okay, this is actually touching one of my customers. But I have no idea about the magic black box that's happening behind the scenes and who does what there and why it's important and stuff like that. And it's really on, I think, engineering leaders in the organization to make those business cases. But anyway, Malcolm, over to you. What are your thoughts on this? Yeah, so our user base, I would say we tend to work a lot with people who are somewhat established but deciding to transition into IAC or they're up and coming and maybe they just created an ops team or something like that.

45:55And a lot of what we find is it's really just this cultural inertia of I'm used to clicking around the UI or we want to get more of a structured workflow. But half of our team is just doesn't want to get their credentials revoked and they'll be really upset about it if they do. So we got to ease into it. So a lot of us, what we see is just definitely cultural and not even just even ones with buy in from from above. just getting the rest of the team in on it. And especially in the larger organizations that decided to adopt IAC, they can have groups in just other countries and on different time zones.

46:35And once you revoke those credentials and someone's really upset, you don't want to like deal with that in the morning or you maybe have like a language barrier, a cultural barrier that it's hard to explain what's going on there. So it's definitely, it's like fractal, right? Like we're talking about the Reddit posts and you have some people coming in just saying, hey, I'm trying to do this, but I can't figure out how to do it. So if you're converting to IAC and you don't have someone who can whip up those examples really quick when someone is, for example, trying to set up their networking layer, the person's just going to get frustrated and click around and be like, fine, it's done.

47:11I figured I didn't have, I did in half a day and their manager's like, great, that's awesome. Rather than spending three days working through how all the providers work. And it just takes a lot of time. That's why I think what Corey said about just start the new things and how you want it to be and go from there is absolutely fantastic advice for anyone doing this. It's just too big of an elephant if you look at all the things you have right now. Yeah, you don't have to boil the ocean. You can, you know, crawl, walk, run, essentially. Well, Corey, Malcolm, this was awesome. Thanks so much for being here.

47:40Yeah, we appreciate it. Yeah, thank you. It was awesome. Cheers.

47:50Thank you.