CI/CD with Robert Erez

14:58Rob Erez:It's fascinating because I always assumed that Kubernetes has won the cloud native space and Skyper scalers. Can you tell them a bit more about how it's being used on-prem? Some interesting stories. You must have seen some, because you said that you're working with companies who are managing large on-prem Kubernetes or interesting situations.

15:20Robert Erez:Yeah, this is one of the nice things about working at a company like Octopus. We talk to and deal with so many different customers and everyone's doing things a little bit different, although they've got slightly different needs and requirements. And you kind of get exposed to a lot of different problems and patterns. And it's easy to sometimes to get lost in what people are talking about in conferences and everyone's saying it's all about cloud and this is the best practice and you should be doing this. And the reality is everyone's kind of got their own little problems and they just want to solve them the way they kind of need to solve them.

15:49Robert Erez:And so some of our customers, in fact, a lot of our customers will run Kubernetes kind of, you know, quote unquote on-premise. So for example, I was actually talking with one.

15:58Rob Erez:When you say on-premise, can you just be a bit more clear? Is this a data center where they're like renting and co-locating? Is this actually like I have my own data center? Or is this like I actually have my own machines in my closet?

16:10Robert Erez:Yes and yes, I guess. What? Even in a closet? I was trying to joke there. I'm sure there are still teams out there that are running, you know, the core accounting tools and etc. under Steve's desk. But even when we talk about, you know, small computers, some of our customers have Kubernetes clusters basically in their point of sale systems. So they have hundreds and hundreds of stores and they have little Kubernetes classes that essentially run in them and each one's independent. And they run into their own problems with that because particularly at scale, when you've got thousands and thousands of clusters and these customers are following various GitOps practices, etc.

16:49Robert Erez:where they're pulling the actual state from a Git repository. So the Git repository itself becomes the bottleneck or they start getting throttled. and so they have to sort of resort to other mechanics to try to sort of mitigate and work around that. I was talking to another one of our customers actually just the other day at Kudcom there who they are deploying, they've got Kubernetes clusters running on research vessels and those research vessels, as in boats, as in ships, like on the ocean. That's right. I'm not going to pretend to know exactly what they're doing on those ships. We didn't quite get into that detail but they've got kubernetes clusters out out in the open sea right which is apps given kubernetes name the problems they run into though are a little bit different right so for them um you know those boats might be out at sea for i don't know weeks months at a time whatever that might be so when you want to do a deployment that the ship's not available so when that ship comes back into port it needs to get the update right so that we're talking through how you would how you'd achieve this right and how that process would work this is super interesting and i love how

17:51Rob Erez:you kind of get a peek into so many different types of teams through the fact that you're talking with them about how they do the deployments, but you probably see some other things that they're doing or things they're struggling with. What are some trends you're seeing across the industry in terms of this wide range of companies you work from startups to finance companies to these research vessels?

18:13Robert Erez:Yeah, I guess one of the big trends these days is a lot of focus on GitOps. So GitOps is... What is GitOps? What is GitOps? That's a good question, Gugay. let's take a step back for a minute so you know we mentioned we talked about kubernetes earlier um and we talked about the fact that it's kind of got this internal continuous reconciliation process where you say to the cluster uh please spin up you know five pods and um it takes that desired state and ensures it always sort of is true in the in the world and so there was a lot of products around that were doing similar thing you know terraform does that for infrastructure etc um and a bunch of people started wondering why can't we sort of take that process and pull it back further so that not only is kubernetes just dealing with desired state but we can pull it sort of directly out of git and so you know i can as as an engineer make changes to that that git definition that desired state and i'd have some process that essentially pushes that to the cluster and ensures that it remains in in line with what i'm asking what i'm expecting and so the term GitOps was coined by Weaveworks in, I think it was 2017 or so.

19:19Robert Erez:And as a general practice, it sort of started picking up steam, particularly in tandem with Kubernetes, because at its core, Kubernetes is very declarative, right? Later on, sort of in the early 2020s, it was kind of formalized a bit more, and there was sort of four key pillars of GitOps. The first being essentially declarative, you want your state to be declarative. So this is the idea that you want to define what you want the state of your infrastructure to look like. This is to basically make things a lot, I guess, simpler to understand what the state of the world is going to be when a deployment takes place.

19:55Robert Erez:So if you think about deployments that are a bit more imperative, that has sort of a process, the end result is sort of the result of multiple steps. But when you're wanting to just update some infrastructure, that desired state kind of works really well, particularly in the Kubernetes space.

20:12Rob Erez:And then in GitOps, the desire state will be just describing like how many nodes I want or like how many, I don't know, replicas do I want on a database or how many web servers or like load balance or how to be connected, that kind of stuff.

20:25Robert Erez:That's right. Yeah. So it's basically a way of being able to say, I want my infrastructure to have whatever state it is. And then the GitOps agents, the GitOps products basically ensure that remains the case. So they'll keep applying it to Kubernetes. So you've kind of got this situation where Kubernetes keeps its internal status in sync with reality. And now you've got these GitOps tools that take the declarative configuration in sync with what Kubernetes is.

20:54Rob Erez:So they will take the, whatever I put in Git and whatever format I use, and they kind of translate it into something that makes sense for Kubernetes and now Kubernetes can apply it.

21:02Robert Erez:Yeah, I mean, ideally you want it

21:03Rob Erez:as close as possible to what sort of,

21:06Robert Erez:I guess, Kubernetes is expecting. because or it allows yeah that's right and so what you're describing there i guess is the continuous reconciliation and so this is the idea that um these these git ops apps will um essentially as we said sort of take that state and apply it and if there's any drift from kubernetes side so for example someone um you know runs kube control you know delete pod or a delete deployment or whatever the case might be because your desired state is now stored in in git in this case that will kind of self-repair the second pillar of GitOps is that that desired set you've sort of defined should be stored somewhere that's immutable and versioned and so this is the idea that once I say that I want to have this state I want to have sort of something I can point to a pointer and I might be a tag or a commit char or whatever and I want to basically use that to define what what that actual state should be and I don't want that to be able to change, right?

22:05Robert Erez:Because otherwise that kind of defeats half the point. By having it versioned and immutable, it also makes things like auditing a lot simpler, right? You can see the transition of that desired state over time. What's interesting, though, is a lot of people will point to that and go, yes, versioned and immutable, I know what that is, that's Git. I was about to say that,

22:25Rob Erez:because Git definitely gives you versioning, or it gives you commit history. I'm not sure if it gives you versioning, and immutable in the sense that, I mean, the past cannot be changed.

22:33Robert Erez:That's right.

22:34Rob Erez:Or actually can it?

22:36Robert Erez:Because you can rewrite the history. You're right. So depending on how you sort of configure your GitOps agent, you know, you certainly can rewrite history. If you have it pointing at a tag, for example, you can change tags. And so that's why there's, you know, best practices around that, I guess, kind of, you know, wiggle the finger a bit if you're using tags to manage that sort of state. But what's interesting, though, is really nothing in these pillars, and very quickly, the third one being pull versus push. And so this is the idea that your GitOps agent will pull the state from GitHub and put it into the cluster, and the fourth being continuous reconciliation.

23:18Robert Erez:But nothing in any of these sort of pillars actually talks about Git. And I think that the naming of GitOps kind of gets people to already have this expectation that everything has to be in Git.

23:29Rob Erez:I mean, why would you not have that expectation?

23:32Robert Erez:That's what I assumed. That's right. I think the problem, though, is not everything should be in Git, right? So you've got this constant kind of conversation within that community about, you know, where do you put secrets, for example? Not in Git. We know that, right? Do not put it in Git. And so that's the thing. So, you know, there's been all these solutions to try to put it in Git. So there's things like sealed secrets where you encrypt and put it in Git. Sounds like a terrible idea. But I guess what's really, this is highlighting is the reality that some things don't need to be in Git, right?

24:03Robert Erez:As long as you can have this sort of control over the versioning or immutability of it, then that's completely fine.

24:10Rob Erez:And then the trend around GitOps is what you're seeing that a lot more infra teams are moving from, okay, a few years ago, they might have just like made definitions for Kubernetes, and now they're moving over to GitOps. So saying, okay, we'd like to control infra in a tool in a way that's described, that's in version control. Is that the trend or what is the trend around GitOps?

24:32Robert Erez:I guess it's more just the trend of the growth in general of GitOps in enterprises, right? So not every company out there is using Kubernetes today. And as they sort of approach Kubernetes and they're looking at, well, how do I perform the deployments? How do I manage that process? GitOps becomes the sort of de facto process. And to some extent, it is giving rise to this idea of using it to manage other things outside of Kubernetes. And there are a few examples of projects and experiments that will use things like Terraform. And there's a continuous reconciliation service that keeps your actual state outside in sync.

25:09Robert Erez:At the moment, it's really about the focus is on, I guess, Kubernetes is the core place where it lives. And I guess it's more the growth of Kubernetes itself means that GitOps is coming along for the ride.

25:21Rob Erez:And you mentioned enterprises, which means like these large companies with oftentimes thousands of people or in regulated environments. That's what I think of enterprises. Are you also seeing smaller teams pick up things like GitOps? Is it like everywhere or is it more there's certain types of teams that seem to be just more interested in? That's a good question.

25:39Robert Erez:So I guess sometimes what we see is a lot of people go to conferences or they read blog posts and they hear that GitOps is what you should do. So I guess what I want to point out here is GitOps is potentially not necessary for all locations, all teams, right? There's certainly a bunch of benefits to it. But the reality is there's some things you need to do outside of just GitOps. You might use GitOps principles in parts of your process, but some of this absolutism, I think, that sometimes exists may not be necessary. So there's often a bunch of other processes you do around your actual sort of, you know, quote unquote deployment.

26:16Robert Erez:So things like maybe you run smoke tests or maybe you want to send a notification when it's complete or maybe you want to do a database update or something like that. These kind of steps don't really lend themselves very well to kind of this declarative everything is in Git kind of process, right? And so that's why you get things like Argo workflows and rollouts and things come out to try to kind of GitOpsify this process. And that works for some people. but the reality i guess is that um i think some people get really hung up on this idea that that everything is git so therefore they've found the tool i don't you know and so therefore everything is a nail yeah i think that's just not the case and this is the thing like talking with with customers when we go through this process of you know you can use uh git ops in octopus um and you know we've got a bunch of support for various um mechanics that integrate well with kubernetes and Argo but there's a bunch of other sort of operations you do around that process that that doesn't need and when you talk to them about it you know they realize that what they're trying to do is ultimately just ship software so again that difference between what you um hear when you talk at conferences and things where you know everything is everything is good and everything must be you know in in this particular format or whatever the case might be the reality is for most customers they're just trying to ship software right and they don't care what name you give it if it's GitOps and it works end-to-end and solves everything, good.

27:41Robert Erez:If they want to use GitOps as part of the process, but then have other mechanics that are more sort of imperative, then good. It's just sort of the reality of, you know, there's tens and tens of thousands of companies out there in the world that are doing software delivery. And not all of them are at conferences, and not all of them are at the forefront, I guess.

28:00Rob Erez:As Rob says, most teams don't care whether you call it GitOps or anything else. They just want to ship software and know that it works. Our presenting sponsor, Antisyses, does exactly that. is lets you ship knowing that it works. Antithesis goes beyond code review. It runs your whole system inside a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, Antithesis doesn't only find bugs, it gives you a perfect reproduction of every issue. I know this sounds like science fiction, but it's actually hardcore engineering under the hood.

28:33Rob Erez:JaneStreet, Fly.io, and the Etsy, the community ship agent-written code with full confidence because they know it's been verified by Antithesis. To see more case studies and details, head to antithesis.com slash pragmatic. That's antithesis.com slash pragmatic. I also want to mention our season sponsor, TurboPuffer. TurboPuffer is exactly the thing that just works. A vector and full-text search engine built on object storage. Fast, cheap, and extremely scalable. No exotic architecture required. Here's something I find interesting. The teams building the smartest AI products out there, Cursor, Notion, Cognition, and Tropic, They all run on TurboPuffer, but why?

29:10Rob Erez:Let's think about it. An LLM without context, it can feel pretty dumb. I can still remember shortly after ChatDBT launched early 2023, how it felt both incredibly smart, but also frustratingly stupid. If you asked it a question that was outside its training data, it just made things up. Fast forward to today, and the models and their tools for retrieving context are much better, but hallucinations still happen frequently. Here's a typical way to integrate TurboPuffer with an LLM. Plug it behind your search tools, get fast responses and relevant context back. The neat thing is how it gives you the perfect blend of low-cost storage, fast retrieval, and a bunch of different search tools like Vector, Full Text and filtering.

29:49Rob Erez:You can afford to index billions of documents and then you can query it with different tools to get the most relevant handful of documents in milliseconds. This is how your LLM feels smart. It gets the right context really fast without blowing through a bunch of tokens. Of course, you can use TurboProffer to search anything, not just code. If you're building AI products, check out TurboPuffer at turbopuffer.com slash pragmatic. With this, let's get back to Rob and talk about progressive delivery. Yeah. Another trend that we talked about just before is the rise of platform teams. Can you talk about what you're seeing?

30:22Robert Erez:So platform teams are kind of, I guess in the past several years, they've become this sort of new standard organizational structure to help teams manage their, I guess, deployment workflows, a bunch of the infrastructure around it. And it's kind of come out of this evolution of DevOps, right? So, you know, we mentioned before in the old days, you'd write a bit of code and you'd throw it over the wall to the ops team. So it was dev teams and ops teams. This was like in the 2010s, 2000s. Back in the long, long ago. And then DevOps became, you know, the practice that everyone sort of realized that actually we want to have the engineering teams be involved in and have ownership of part of that that operational um process idea being you know you get faster feedback loops you are able to kind of if you feel the pain you you sort of fix you know it's that saying you know you fix it you ship it um you know we've all kind of heard that and so um a lot of teams you know took that to heart that's that's good great good practice but as things start to scale up what you'd find is that there would end up being like a devops team again and sometimes sometimes it's to another ops team and so there'd be the separation of of um development and devops and it kind of goes against some of the principles of what devops was you know trying to destroy but not only that these teams then um end up um having lots of different ways of doing their deployment so you know you've got every single you know how a bunch of application teams and they've all got slightly different requirements and all building it from scratch and so you'd end up with these these teams either whether you had the devops teams or was still within the application teams where um there was just this this um large number of different ways of doing things right and that becomes difficult at scale so um you know you can't really move between teams and by

32:13Rob Erez:scale you mean typically when there's a lot of teams right that's the easiest yeah that's right

32:17Robert Erez:if you've got lots of lots of teams and each one is kind of owning that process end to end you know you sort of get this bifurcation of processes and not only that the application teams themselves start kind of getting this context overload, right? They now need to think about what's best practices of the different cloud tools. Yeah, and there's a job of devs

32:37Rob Erez:rarely wants to configure the deployment scripts and test them and testing is hard. You can't stop your unit testable. So it's now a different job. I remember when I was on earlier teams where, you know, like typically on a mobile team, like you have a mobile team of five people and one of them one of us had to kind of specialize in Jenkins configurations because Jenkins is oftentimes the or used to be the mobile CICD and it's kind of like half a person dedicated to that and it was

33:04Robert Erez:more like draw you know like we had to draw a stick on who's going to do it because we want to

33:08Rob Erez:build stuff.

33:08Robert Erez:You want to write code right you just want to focus on writing code and so if you're spending a bunch of time sort of managing infrastructure and pipelines and things you know that that's no fun for anyone and so platform teams have come about as a new way of solving that problem where it's different to kind of you know this idea of a DevOps team or Ops team that kind of own the whole process they they more sort of define best practices and they provide a ideally a self-service mechanism where application teams can essentially use often what's called as an IDP an internal development portal and they'll be able to essentially self-service and you know maybe they want to spin up a new project and they're able to use a template that the platform team have generated and so the platform team are able to sort of create these standards throughout the throughout the company and they can be responsible for sort of i guess the definitions of those processes and the best practices and how to achieve that but the ownership of the the actual running operational sort of element is still within the teams right so they still get those benefits of you know devops being close to the close to the real code and and feeling the pain if there's a problem and etc etc etc but they don't need to spend all that time becoming experts in you know all the different ways you can deploy the software they've got and so this has become really common now where particularly as you sort of get to a larger size platform teams are a great way of solving that problem now that's not to say that every company everywhere should have a platform team i mean if you're a small company sometimes it's you you've just got the app steam and they sort of are doing, you know, quote unquote DevOps.

34:45Robert Erez:But this is certainly something that as you sort of start seeing larger organizations with multiple teams and multiple projects, these platform teams are a way of basically bringing some sanity and control and focus, I guess, to the whole space. One trend across the industry, of course, is AI.

35:03Rob Erez:It's hard to see any teams where devs are not using AI agents specifically to code. your product managers will be using these things and of course we have a lot more code produced as a result when it comes to cicd systems what are you seeing changing there because of ai

35:23Robert Erez:this is the this is the uh the elephant in the room right though that is ai affecting sort of the reality is i think to be honest it's it's still very early i think what will happen is the impacts of CICD are really tightly coupled to how development teams end up using AI. So there's going to be some sort of like a, I guess, a lagging process there. But we're finding a lot of people, a lot of teams are starting to use AI in the development process. And so we're starting this process of going out and looking and talking to customers and learning what's the way that they're handling AI in their teams and their application teams.

36:00Robert Erez:and then how we can best leverage sort of the CI side to support that. But then in addition to that, use AI within the pipeline itself, again, in the right place. So one of the things we've been, I think, pretty keen on at Octopus is this idea that, you know, at KubeCon, we were probably one of the few companies there that didn't have, you know, AI plastered all over it. Like we tried to be very, you know, that's what gets the sales, right? Yeah, that's what you send out now these days. That's right, by not having AI. I mean, we've got AI in Octopus, but what we've been trying to do is think about, well, how do we actually use it in a way that's actually useful for our customers, right?

36:42Robert Erez:For engineers, et cetera. And so we've been slowly adding capabilities within Octopus to provide AI support, whether it's a MCP server, whether it's a recovery agent that can review logs and tasks and all that sort of thing. But that's within the product itself. Some of the bigger changes will depend on, like I said, how actual application teams use AI. What I think, you know, we're talking about, we'll find is there's going to be a lot more velocity. I think that's one of the big, big changes, right? There's just going to be a lot more code coming through. I think one of the questions is, okay, what does that mean for your pipeline?

37:19Robert Erez:One of the things you often talk about when, you know, there's a human element to the pipeline is speeding up the cycle to get that feedback quicker. If you've got engineers sitting there waiting for their code to run tests so they can get back to it and fix it, the shorter and shorter you can make that feedback loop, the better it becomes because they don't need a context switch, et cetera. I think in a world where the majority of your code is being developed by AI, that becomes perhaps less important. If you can kick out your build and test process and it takes 30 minutes versus 20 minutes, does it really matter if the engineer's already long gone, moved on to the next problem and the actual AI agent itself can kind of babysit the process and review the problem that came up and issue a new fix.

38:04Robert Erez:I guess there'll be a de-emphasis, I think, on some of the speed of the pipeline itself and more on increasing or decreasing risk, right? The risk that comes from having AI agents generate code. And so exactly what that process looks like, I guess, remains to be seen. I think what we'll see a lot more use of is things like progressive delivery and I think particularly feature toggles are going to be a really common tool in the tool belt of application teams, partly because it allows you to ship that code as fast as you can or as fast as you want, but manage the rollout of the actual feature set or changes sort of independent of the deployment.

38:40Robert Erez:So it decouples your deployment from your release. And so in a world where we've got a lot more AI agents generating code and being involved in perhaps part of the build process, those agents themselves being able to use toggles to react to it quickly, I think then become a lot more important than perhaps what we see today.

38:59Rob Erez:Can we talk about progressive delivery, what it is, and what are the most common ways to de-risk getting your code or your software out there?

39:11Robert Erez:Progressive delivery is the next evolution beyond continuous delivery. With continuous delivery, it's this idea that I've made a change to the system and I want to ship it to dev or staging, or typically if it gets to production, sort of in one hit, right? With progressive delivery, what you're trying to do is basically release those changes in a little bit more of a controlled way, typically through things like a canary deployment. So this is where you might deploy some subset of your instances that are out there. So what is a canary? What is a canary? Canary deployment is, this is New Zealand, basically.

39:49Robert Erez:New Zealand's our canary. So this is, as we said before, this idea where you select some subset of your customer base or whatever that might be, and you would typically route traffic to a new instance. So you'd ship, you know, you've got version one running and you want to release version two. You essentially ship version two side by side and you might use, you know, most common one would be some sort of network traffic manager to route some percentage of your traffic to that new instance. and you gradually roll that up. Typically, you know, as you do this process properly, you should have a fairly mature observability mechanism in place to see that, you know, you can roll up or roll down.

40:30Rob Erez:And I guess this whole thing comes from a canary in a coal mine, right? That's right, yeah.

40:34Robert Erez:So the idea being that, you know, in the old days when you'd be in a coal mine digging away and it would release, you know, all sorts of toxic fumes and things like that, canaries were a lot more sensitive to it. So they have a little canary in a cage and if that canary sort of died, I guess, got knocked down.

40:51Rob Erez:I think the canaries, as I understand, they were like chirping. And then when it stopped chirping, well, it also died. Oh, okay. So, all right.

41:01Robert Erez:Same ending, but just, you know, a nicer way to go out. They need to get out. Yeah. So it's this idea that you get that advanced warning, I guess, that, you know, rather than you getting knocked out by the toxic gases, et cetera, you know, you can get out of there sooner. So it's that same principle, I guess, brought to the software. There's various other mechanisms like blue-green deployments. So you've got your first version there still receiving traffic, and your second version is up and running, and you can now do some tests against it, validate it. Maybe you've got sort of the IP details to access it directly.

41:34Robert Erez:You can basically validate that it's working. Sometimes there may be a way of avoiding cold starts and things because that process may need to initialize a bunch of stuff. But then when you've sort of done that validation, and you're ready, you can essentially swap traffic around. So all the new traffic goes the other. In some ways, it's like doing a canary, but straight to 100%, but you're doing a bunch of validation sort of on the side before it actually reaches customers. In my view, probably the more useful progressive delivery strategy is feature toggles. So this is the idea that you've got some sort of...

42:07Robert Erez:Feature flags as well, right? Feature flags, feature toggles. Yeah, that's right. Yeah, often used interchangeably. So this is the idea that you've got, you know, know some sort of variable in your system and it's linked to typically some sort of external service and through the state of that particular variable being sort of true or false on or off you can essentially have different code paths essentially take effect and there's a bunch of benefits that feature toggles have over i say canary releases particularly for application delivery where you know the the your unit of change with a feature toggle is is very granular It can be, you know, single lines of code.

42:44Robert Erez:And so everything else remains the same. And all you're doing is tweaking that single line of code. With a canary or any sort of versioned sort of delivery deployment mechanism, your unit of change is the entire app. So if you've had, you know, 20 commits since the last sort of release went out, then you're essentially testing all 20 things in that one hit. Your ability to sort of target the actual customers, it's a lot more precise when using feature toggles. So you can use all sorts of complex rules and say that, I don't know, everyone from Germany who has this particular product in a basket has this kind of experience.

43:21Robert Erez:And that's, you know, really hard to do via network traffic rules, right? The other is your ability then to actually roll back. So to sort of roll back from a canary, hopefully you're still in the process where you're sort of going through that canary process and you can roll it back. That could take, you know, minutes. Maybe you have to redeploy the whole old version. That could be minutes or more. With a feature toggle, you know, you can do that in seconds. That's pressing a button and it happens immediately. Not only that, but you've kind of, you've got more control, I guess, on when you sort of do that.

43:52Robert Erez:So with a deployment that you're doing via a standard versioned release, you're sort of tied to when that deployment takes place. Because when it takes place, that's when essentially your new feature is available. And as an application team, that means you need to know about exactly when it's taking place and make sure you're watching the logs at that point. And maybe you and 10 other teams who are shipping things at the same time are all doing the same thing. Whereas with feature flags, you've basically got control over when that takes place. So you might ship the assemblies and that sort of thing on the Monday, but you release your feature on Tuesday when you come in and you've got the logs ready and you've kind of reviewed what the next steps are.

44:32Robert Erez:So it really makes things a lot easier to decouple releasing a feature from deploying software. You know, version deployments through Canary, etc., they're really useful, particularly if you're doing like infrastructure type changes where there is no kind of application toggle that's relevant there, but you want to violate some changes to your infrastructure or your process, or potentially, you know, things like things that will involve schema changes. and schema changes are the big, the big difficult part. Database schema changes, right? Database schema changes. This is the big problem in any, like to be fair, in any progressive delivery.

45:09Robert Erez:And this is why, you know, the question always is, are you ready for progressive delivery? To do schema changes, I guess this is the point that application teams kind of need to be really mature. And I don't mean mature in terms of, you know, not telling silly jokes, but mature in terms of understand all the problems that are in place with this and know how to release these, of changes in a gradual controlled fashion and do it over multiple stages that you know ironically is actually quite hard for us um and octubus because our software is both sas hosted so we have a sas offering that customers can use and we have an on-premise version and it's kind of because we have both both sides um we kind of have the best and worst of both worlds you know in the cloud system if you've got a sas product you have complete control over what versions go So if you want to do an expand and contract, you can stage the whole process.

46:01Robert Erez:You know that it's all been updated before you kind of move to the next stage. On the other hand, for a self-hosted application where they go and they install it on their own infrastructure somewhere, you don't know what version they're running and what they're coming from. So they might upgrade from version one straight to version six. And so you're not really forcing them to go through that expand and contract phase. On the other hand, you know, they've got a lot more control over when they upgrade. and so you can kind of be a little bit more deliberate about you know making sure that they do backups before they change and and you know maybe the down maybe they can manage that migration and accept a little bit more um downtime during migrations and updates and things like that then would actually be you know acceptable in a sass product so one thing about you know

46:43Rob Erez:we talked about progressive delivery and you're kind of doing this to avoid surprises you know If a regression goes out, a new bug or something doesn't work, you kind of want to catch it early. Hopefully only a few customers have experienced it. Or even if it's not 100%, you kind of... And you have a way to go back. All you do is, if it's a feature flag, you hide it. If it's a canary deployment, you go back to the other one. But there's also this thing where when things do go wrong, at some point you want to do a rollback. Can we talk about how have you seen rollbacks done well? And what does it take to actually have a real rollback strategy?

47:24Rob Erez:A bunch of people talk about CICD. Some people talk about feature flags. I don't hear too much chatter about rollbacks.

47:29Robert Erez:Yeah, rollbacks. This is always a spicy one. We get a lot of customers who say, why don't you have a rollback button? I want to roll things back. Why can't we roll things back?

47:37Rob Erez:As in the deployment software, like Octopus or anything else. They're like, okay, if it can deploy, I want to do checkpoints and just do a rollback.

47:46Robert Erez:That's right. How hard could it be? Just do what you did before. How hard could it be? Tell me. Well, this is the problem, right? So in a completely stateless system, that's pretty straightforward. If you've got a completely stateless system and this is something that GitOps is really good at where you'll have that definition. So it's somewhere in repo. If it's completely stateless, you can do a Git revert and push it and it'll go back. The reality is for most systems out there, you've probably got some state. state being databases. It could be any sort of information that you can't necessarily just kind of undo, I guess, because if you roll it back and now you've got your code talking with the schema of the database that's not in sync, you can provide schema.

48:29Robert Erez:If you've got a schema migration, let's say in a normal deployment, you can provide alongside that a secondary sort of anti-migration that kind of undoes the change. But again, that's not always possible. you need to do with what are you going to do with that data state. That's a lot of work. We've gotten pretty far in basically trying to advise customers that you want to avoid ever talking about rollback. It's always roll forward. So if there's a bug, roll forward. Tell me more. Yeah, get your change in as soon as possible. This is where fast feedback loops are important, right? This is what the hotfix processes are for, right?

49:02So we all know that in a standard process,

49:05Robert Erez:you want to go dev, staging, prod, and maybe you've got approval processes and it slows down etc but if you've got a significant significant bug that you need to kind of quote-unquote roll back sometimes the safest thing to do is actually make a hot fix to that that version and push it out sort of as quick as possible and your bottleneck might be the the build pipeline or whatever but depending on sort of your appetite for risk there you can resolve that sort of a lot quicker now obviously if the failure itself is just from some mechanism in the deployment process itself or somewhere further down that chain then your your time to recover is going to be a lot quicker.

49:40Robert Erez:But it's this idea that, you know, if I've got a failure in version two, my rollback isn't to go to version one, it's to go to version three and make sure I've got that fixed in version three. It's the sort of thing that, you know, when we talk to customers and some of them go, yeah, we roll back, you know, we roll back all the time if there's a problem. And then when you ask them, what do you do if you've got a schema change? They kind of stop and realize that they've never, it's just sheer luck that they've never sort of run into that, right?

50:07Rob Erez:Is it fair to say that you want to roll forward if it involves business logic or something that is not stateless? Because if it is stateless or if it's application logic, you know, you have a code that says if this else then and you realize there's a bug there, you can just revert it as long as it doesn't touch the schema or the data.

50:26Robert Erez:Yeah, I mean, in an ideal world, you're reverting is through a feature flag, right? That you click and you're essentially reverting by changing the code path. And this is why I always say feature flags are kind of a nice tool to use for doing this progressive delivery because, you know, it's just as easy as to roll out that feature. You can typically roll it back. Now, you're still going to have some of those problems with schema issues, et cetera. You know, if you're making a change and you've got parts of your code path that expect one and not the other, you're going to need to account for that.

50:55Robert Erez:But you can even account for that inside the feature flag. That's right. Yeah. Yeah, so that's the way you sort of ideally sort of manage that. So that within, regardless of which path you go down the feature flag, it's kind of self-consistent with whatever version of the actual database schema that's out there.

51:09Rob Erez:So I guess the more feature flags you use, the fewer surprises you might have. But it's a bit of extra work both to build and also to remove.

51:17Robert Erez:Yeah.

51:18Rob Erez:You get stuck with still feature flags all across your code base once you start to use it a lot.

51:22Robert Erez:I saw this at Uber. Yes, yes, a hundred times yes. So when you're adding a feature toggle to your app itself, so we at Octopus, we obviously use feature toggles in our code quite a lot. And we use Open Feature as like the framework, the SDK to interact with it. But we essentially have built a wrapper around it where the toggle itself within the code is sort of, we provide some details about which team owns it. And that team sets an expiry on it. Now the expiry itself, when that time passes, nothing bad will happen. But through parts of the CI process, if that time has passed, we can send a notification to that team and say, hey, it looks like this toggle is no longer used.

52:02Robert Erez:So the specific mechanics don't matter as much, but it's more a matter of making sure that, you know, if you're adding feature toggles, it's really easy to forget about it because you start rolling it out and you kind of forget about it. And, you know, you want to keep it in there just in case for a while, in case you need to roll it back. and having the ability to understand how long a toggle has been there is kind of a key part of helping to maintain that hygiene. Now, the reality is even at Octopus, we've got a bunch in there. I know I've got a bunch in there that I'm sure if I was to log in, I'd probably get a bunch of notifications to remove.

52:35Robert Erez:When we use that gardening metaphor in code, this is one of those sort of operations. This is weeding, right? You need to just kind of keep on top of it. There are some mechanisms around, even in lieu of the AI side, which will, you know, ideally if you're using feature toggles, you've probably got a bunch of observability and metrics and logging around it. And there are some tools out there that will allow you to keep track of when the last time a toggle was kind of evaluated. And that kind of gives you that signal. Similarly, you know, you might remove it from the code because typically when you want to remove a feature toggle, you want to remove it from the code first before you touch your actual sort of toggle system.

53:09Robert Erez:And so having a mechanism so that once you remove it from the code, um you know it might take two weeks before it makes all the way out into production so you don't want to delete it before then by that time you've kind of forgotten about the fact you removed it oh yeah um and so having mechanisms that will keep track of that change i guess going through the system um and when it reaches the environment where um you know production where it's actually being used can kind of show okay that code's gone out that's you know removed the toggle it's it's fine and safe to actually remove the configuration because you've got that feature toggle information in two places, right?

53:40Robert Erez:You've got it in the code and you've got it in your platform.

53:44Rob Erez:Can we talk about how developing environments evolve? We talked about CICD, but I'm interested more in, you know, you go from like, you have one environment, later you might have staging or something. And what evolution have you seen across all the teams that you work with, all these hundreds or thousands of teams?

54:04Robert Erez:Yeah, I'm not sure if there is one particular pattern there. I mean, I think most common is dev, test, prod.

54:14Rob Erez:So these three different environments.

54:16Robert Erez:Yeah, and I mean, even that I think is probably a gross simplification of all the different kind of mechanisms.

54:21Rob Erez:And dev meaning my local machine.

54:24Robert Erez:Dev in the case of CD is often like the first point of integration. So it's kind of test often. Customers will keep tests kind of reasonably in sync with, let's say, production or some sort of sanitized data source. so that way that whether it's the QA testers or the product team or whatever can go and review the code dev is almost like the first point of integration that is it actually is the deployment process just at its core actually working or is anything fundamentally broken at all I think more and more now we're finding that dev is less useful in that respect and what we're seeing is more the growth of things like ephemeral environments and so this is the idea that you know I as an engineer I'm running some sort of feature on a feature branch, and I want to kind of evaluate that it's actually doing what we're expecting to do.

55:15Robert Erez:But not only that, I want the rest of my team to be able to see it working. And if I've got it running on my machine, it's not exactly easy to give other people access, I guess. And then I may want to completely context change, move on to something completely different. So ephemeral environments is this idea that from my branch pre-merge, I wanna spin up a whole environment, essentially from scratch, ideally with whatever dependencies are required to sort of run this particular component of the building. And then I wanna basically deploy my app into that as if it was a normal full-fledged environment.

55:53Robert Erez:As once that's available, I wanna sort of have access to, you know, if it's a web app, maybe it gives me the URL and I can poke around it and hand it around and other people can kind of evaluate. and then the moment i kind of merge that pr tear it down again you know it's quite common to have multiple test environments because you know i've got a lot of stuff going through my pipeline and i've got three testers so let's have three environments so they can all sort of have one at once or often you'll see a single test environment and a bunch of tests and they all kind of need to collaborate to see who's got access to the system at the moment etc etc whereas with the ephemeral environments it doesn't roll off the tongue with ephemeral environments you can essentially have a full-fledged deployment per feature.

56:37Robert Erez:And so again, that's about speeding up that feedback process, right? Again, all of these processes are all about speeding up that feedback process to catch those failures or issues or bugs or whatever sooner.

56:50Rob Erez:There was a time a few years ago where cloud development environments were really talked about a lot, which was the idea is as a developer, you have an environment spin up in the cloud, your let's say your Visual Studio code connects to it or maybe you just log in online and it spins up all the dependencies oftentimes done with containers which reminds me of this as well and there's also like preview environments but somehow it feels that both that discussion and this one kind of died down maybe it's AI, maybe it's something else but I mean the technology is there, right? We have containers, you can package things together I'm sure it depends but it's all doable.

57:28Robert Erez:Yeah, it does get tricky. it's this is again one of those sort of things that's really easy to talk about um for simple cases it it can get tricky when you know what if i've got more than just a single app in my kind of quote-unquote environment and how do i make sure it's got all the data i need to validate so it can get tricky so or if you have a bunch of services that have state that's right exactly so um there are sort of complications that it does bring but the i guess the the benefits that you get as an application team, particularly application team where you've still got engineers writing code, is sort of speeding up that feedback process, I guess.

58:05Rob Erez:Well, now with AI agents everywhere, that's even better because in a sense that if one of the best ways to validate, you know, we have code reviews and AI agent generates and you look at the code, but isn't it not better to just confirm that this thing works, especially

58:19Robert Erez:when it has the UI? That's right. I think even in that world where you've got AI agents kind of building the code and validating the code, any sort of scenario where you want that AI agent to kind of validate what it's done, you're essentially talking about ephemeral environments, even if it's not exposed to people because it's doing its own testing and poking around in whatever shape or form that it's doing. That still is, I guess, one of these kind of environments, right? So it's ephemeral, it spins up, you've got some sort of provisioning process. And then ideally, once the job's down, and you kind of tear it down.

58:51Rob Erez:I'm interested in learning more about the reality of operating a large infrastructure platform. And, you know, one big one you're working on is actually Octopus Deploy's SaaS offering. How does that look like? And what are the challenges of, you know, like running something where you're running all of these deploy processes, all these CD, you probably have a bunch of different things. What is it like?

59:13Robert Erez:So at the moment, I'm not on the team that sort of builds that, but I can give some of the context, I guess, from history and kind of context there. Originally, when we first sort of decided to sort of provide a Octopus SaaS offering, I think it was 2020 or something like that, it was all VMs. So every customer would basically get a VM spun up and we would have a virtual machine. Virtual machine, yep. And the Octopus, you know, self-installed, would basically get installed onto that VM and they'd get a whole VM for running workloads on, etc. etc and that was very much not cost effective it was costing us something like 100 bucks per customer per month and they were paying i don't know 20 a month or whatever it was but this whole process was more an experiment to see was there a demand um and to his credit paul was happy to sort of you know pass out the credit card to kind of go through this process to see that is this actually the direction we we want to go is there a is is this something that going to turn into a viable sort of direction for the company because it's a big step right going from building software that you can kind of hand out and people can download and manage themselves.

1:00:21Robert Erez:It was like pretty much self-hosted or like run on your own infrastructure. Exactly. Yeah, that's right. And so the demand was there. So not long after that sort of first experiment, we basically started from scratch again. And I worked with a couple of the other engineers back then to start building it on Kubernetes. And so Octopus itself in that space, we have what we call kind of a reef. So what you find is everything in Octopus, We've always got octopus or nautical kind of names around it. So a reef is basically a way of, it's this cell-based architecture where it contains all the resources that are needed for that particular customer's instance.

1:00:58Robert Erez:Well, some of it's shared, but it's kind of broken down into individual cells. And so a reef will contain, you know, the cluster and Azure database, et cetera. And each customer instance is running now in a pod in that cluster. and so as part of that project that was when um i think i was working on converting it so i could run on linux and inside containers and someone else was building the dynamic worker infrastructure so um there were a couple of us that kind of just um got in and yeah really just got it up and running so that way we could kind of start moving forward and i guess stop losing money fast forward to today now there's an entire team that's kind of backs that and we've got you know several thousand customers on it and we run you know many many thousands of deployments every every every month and so now what we're trying to do is there's a project at the moment to basically make the octopus deployment process itself more resilient so what that means is at the moment when a deployment kicks off a bunch of the the process as a it's kind of a imperative set of steps a bunch of that is stored in memory which means that whenever we want to do an upgrade we need to essentially stop running tasks for some period of time so it can kill their instance and spin another one back up octopus itself at the moment doesn't sort of have zero downtime between upgrades so there's a bit of downtime between that we kind of want to reduce that and get that as close to as close to zero as possible with the realization that you know going from downtime of five minutes to one that's that's just work right that's you know you move things around you can maybe change the architecture.

1:02:36Robert Erez:Going from 10 seconds to zero is a much bigger shift. I'm not sure if and when we'll get there, but yeah, there's definitely this big effort at the moment to make the whole process a lot more resilient to basically improve and reduce the amount of downtime that takes place so we can kind of perform upgrades quicker, etc.

1:02:56Rob Erez:One interesting thing you do is you have a SaaS, but you also have an on-prem offering. What are interesting engineering challenges that come from that? A lot of companies have decided to just like honestly just move to SaaS because now they control everything centrally. I think Jira did this, or maybe they're doing it, which is a well-known one, but clearly it's just a lot more work and a lot more headache to have both.

1:03:17Robert Erez:Yeah, and we touched on one of the big problems here a little earlier is that when we want to push out any updates, you know, to cloud, because we control the whole process, we can push it out. And so we have a sort of a gradual rollout process there because each customer is on their own instance. We can sort of deploy each one individually. And that may take, I don't know, a few days to, let's say, roll out a change. On-prem, though, is kind of another matter. So actually, I was digging into some of the stats around this a little while ago and found it took about 200 days for, on average, 50 % of our customers on-prem to get.

1:03:51Robert Erez:Let's say you ship a new change today. It takes about 200 days for, on average, 50%. It's half a year. But then there's kind of like almost an exponential decay there where it takes 400 and something days for 75%. to get it so just it's kind of this curve where i mean we've got customers that are still running you know versions of octopus from five six seven years ago and so whenever we ship a new change we need to basically make sure octavis will work from version you know 2023.1 to 2026.4 and so there's a bunch more baggage i guess that we have in terms of like um schema upgrades and making sure that that whole process actually is achievable.

1:04:32Rob Erez:But why do you do it? A lot of startups will be like, screw it, let's not support all versions. This even happens on mobile. What's the benefit?

1:04:40Robert Erez:And it feels like you're kind of swimming against the crowd with this one. The majority of our customers are still on-prem. And so this is, you know, you're talking about banks, financial institutions, governments, things like that, where they want full control over the system. They want to run it on their own hardware. Now, they may use their own cloud or whatever to run it, but they want to manage the whole process and be in control of let's say upgrades or downtime or things like that so it's certainly um not it's certainly not uncommon and i don't think that's going away um anytime soon as for the upgrade support um we're kind of going through this process actually in the past couple years we've been getting a lot more i guess confident with deprecating features and things like that and just kind of cutting cutting loose old capabilities and then part of that has come from you know fully embracing feature toggles as part of that process I think we're getting a little bit braver in terms of you know removing capabilities that perhaps older customers may may miss but I don't think that in the long term self-hosted will will kind of go away this is one sort of things again where I think it's it's really common to hear you know everything's in the cloud we're all in the cloud again the reality is there's a lot of companies out there where for them it just doesn't make sense or it's not viable or it's not you know it doesn't meet compliance requirements or whatever the case may be also it's kind of a reminder i

1:05:58Rob Erez:think that you actually might have a lot less competition if you build infrastructure software that also runs on-prem because it sounds like there's a demand where companies are like we want to give you money in order for us to run on-prem and i'm sure some of them would do sass if there's no other alternative but for sass it's it's easier to build anyway so there'll be more competition So if you're an entrepreneur or if you're a software engineer thinking to do a business or start a business, it might give you an edge. Yeah, that's right. It sounds like a lot of your customers, you know, the ones who have not upgraded your software for, let's say, five years on one end, you might say, oh, my gosh, what are you doing?

1:06:34Rob Erez:But they might just be happy with it. And if they keep paying you as a business, those are some of your most loyal customers. You see what I mean?

1:06:42Robert Erez:That's right. And this is the thing, I mean, I remember when I worked in, like when I worked in the previous job that used Octopus or any of us who have any other sort of, you know, software that you've got running, potentially you've got running locally. If it just works, why touch it, I guess. And so it's kind of the bane of our existence because it annoys us if we want to ship the features and give them all these great new things. But on the flip side, particularly for something as critical as their deployment system, a lot of customers, once they've got it running, they kind of step away and go, okay, let's just let it be.

1:07:15Rob Erez:And it keeps happening with AI as well in the sense that, for example, I just read that Cursor, their latest coding model, it's updated, I think, every five hours, which is amazing. It just keeps getting better. However, there are customers who, once you have an LLM and it works for you, you kind of tuned it you have the instructions great but oftentimes what happens a new version comes out of a model or major version and it stops working and I assume that there will be more teams, companies, businesses who are like look it would be worth for me money to kind of pin this thing or to run it on my own and just have it stay as is and then I will decide when I want to change it as long as if it ain't broken don't fix it

1:07:56Robert Erez:that's right and I think to Octopus's credit I think we have a really good history at sort of helping customers even when they're kind of on those older, sometimes to the extent of wanting to say the support team, just they're on old instantly. Tell them to get the fixed upgrade. But support team are, I think second to none in terms of their willingness to help. And as you said, if they're willing to pay us, who am I to say no?

1:08:24Rob Erez:Yeah, I mean, it's a business strategy, but I think it's just a nice reminder that there's not just one size. And like, even though I think SaaS is eating the world and we're hearing it and we're seeing it it's nice to see that it's not just that as closing uh if i'm a software engineer and i would like to move beyond continuous delivery continuous deployment and go into progressive delivery what pointers can you

1:08:49Robert Erez:give me yeah i guess just just start with something right so start with adding one feature toggle it may be scary at first to kind of go oh it's in production if i you know toggle this i'm going to break something production you know it's nice and comfortable to know that you're kind of well to the left of of the running systems and if you ship code everything will be caught by the test but you know if i toggle it what will happen it's kind of like a drug right once you start doing it you don't want to stop and that's that's why we've got this this hygiene problem for things like fiji toggles right it's really easy to add them and actually end up with the opposite problem of how do you and you kind of control yourself how do you stop so i'd say just just kind of start doing it add one and and keep an eye on kind of as you roll it out and you look at the results from And the reality is, you know, I've shipped features behind feature toggles where I've shipped a bug, right?

1:09:34Robert Erez:And it's one thing to ship something and turn on a feature and go, okay, cool, customers have it. It's a very different thing when you do the opposite. If you ship something and there's a problem and you can reach immediately for the toggle and switch it back off. You know, the amount of times you kind of, in the past, you have this kind of panic of, I know I've shipped something. It's, I don't know what's going wrong. And particularly when you're in that state, you know, maybe you've got called up at 2 a.m. because you've got an on-call and you don't know what the next step is to do and you've got a panicked mind and should I build a new thing or do I somehow force a redeployment?

1:10:04Robert Erez:So having the capability of being able to flick that switch just allows you to then calm right down and go, okay, I've stemmed the bleeding. Now I'll come back and reanalyze it and understand what's wrong. So having that capability, once you experience that and realize the value of that, not just rolling things out, but I guess rolling that individual feature back off, yeah you'll you'll want to use it for everything what's one or two books you would recommend and why i'll give two kind of i guess technical ones and and more of a fun phoenix project is still for me a good one this is one that gene kim yeah yeah and um i i can see uh you know yeah you kind of remember that we got that in in in skype this was one that abdella kind of gave to everyone and our manager gave it to everyone yeah and um you know it's it's you know parts of it may be a little bit outdated and you know some of the practices have changed a little bit at its core this idea of as an engineer being involved in that whole sort of operation side of your what you're shipping and and the value that gives to not just the company but to you is amazing so i think that book has kind of a core well yeah it's one of those core foundational ones that sets the sets the the context for everything we talked about today the other one from a more i guess organizational and communication side of things.

1:11:21Robert Erez:Radical Candor by Kim Scott allows you to communicate more efficiently and with more compassion with your peers and other people around you. It's, you know, really common. You know, I'm an engineer, so I know sometimes it's really, you kind of look back on what you said and you feel like, okay, maybe I can be a little bit blunt. Whereas Radical Candor teaches us to think about, you know, you want to have those communications that are both sharing that you're caring and empathetic, but also direct. And, you know, the benefits of that and kind of the inverse of that where you know you perhaps like I said you're very blunt you're sort of being honest about it but you're missing that empathy so I found that book really useful and interesting as I guess not even just as an engineer but as a person working with other people from the more fun side basically anything by Greg Egan he's an Australian sci-fi author he writes some pretty crazy and mind-bending hard sci-fi so if you're really into that i'd say read like diaspora or um charles letter they're the sort of books that actually took a second read to get through and he's he's a mathematician as well so you know he's got a whole bunch of background and mathematics on why a certain certain part of his story goes the way it is he you've heard an entire story on the premise of what if the speed of light wasn't absolute or something like that.

1:12:40Robert Erez:This one premise and it kind of breaks out into it and this is what happens to energy and therefore molecules work like this and da-da-da-da. And as a, you know, I'm a tech nerd, that sort of science stuff really appeals.

1:12:53Rob Erez:Same, same. When sci-fi, there's some science involved. That's actually, I find it way more fun.

1:12:58Robert Erez:Rob, thanks very much. Thank you, Gaget. It's been great. It's been great.

1:13:01Rob Erez:What an interesting conversation. I hope you enjoyed having someone like Rob who has been building and thinking about CICD at scale for a decade. It was such a fun blast from the past story as he talked about how at Skype, our team basically did continuous delivery years before most of the industry caught up and how we did it by quietly shipping new builds to New Zealand every week, using this as our canary country. It's a reminder that a lot of modern software practices were already being run in the wild by devs who just want to ship software faster than our change advisory board will allow us to do so.

1:13:30Rob Erez:One other thing I took a note is Rob's take on rollbacks. Lots of engineering teams talk about rollbacks as if they're a safety net. But the moment you have a database schema change in the mix, what's safety net? Rob's advice is to roll forward, not back, and use feature toggles as a way to turn features off or on. This is also a reminder that investing feature flags is usually really helpful. But if you have feature flags, be sure to clean up after them after you've rolled them out. Otherwise, they become a big mess. Finally, a part where I learned something new was on GitOps. I've always assumed that GitOps was about, well, Git.

1:14:04Rob Erez:But as Rob pointed out, none of the four actual pillars of GitOps require Git at all. The four pillars are number one, declarative, number two, virgin and immutable, number three, pulled, not pushed, number four, continuously reconciled. The name GitOps has caused the whole industry to get a bit dogmatic about putting everything into Git repo, even things like secrets, which absolutely should not be there. Rob's take is that most teams just want to ship software. If GitOps helps with that part, great. But if a more practical process works better, just use that. Do check out the show notes below for related to Primatic Engineer deep dives on backend technologies and other related topics.

1:14:38Rob Erez:If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks and see you in the next one.