Voices of Oxide (Interview)

27:02They actually make different amounts of money. They're the one corner case because salespeople, we have an incentive thing where the more they sell, the more money they make, which they were excited about and we don't have a problem with. So, yeah, I do think it helps with the sort of sense of like, we're all in this together, which is good. 100%. Yeah. Well, we'll let you get back to it. All right. Thanks for chatting with us. It's been awesome. Yeah, my pleasure. Thanks, Cliff. Great meeting you all. Appreciate it.

27:33well friends the news is out our friends over at code rabbit code rabbit.ai they've raised a massive series b and they've launched their cli reviews tool it is now out there i've been playing with it it's cool the bottleneck is not code the bottleneck is code review with so much code happening so many people coding now so much code being generated and so many things competing for developers time and attention to maximize code review still remains a bottleneck but not anymore code rabbit cli code reviews code reviews in your pull requests code reviews in your vs code and more teams now have a true answer to what it means to code review at scale code review at the speed of ai and code rabbit is right there for you you'll learn more at coderabbit.ai we'll link up their latest blog announcing their series b and their announcement of their cli review tool again coderabbit.ai okay friends next up is dave pacheco talking about oxides update here we go What is the update on the update?

28:56Well, let's see. We're working on shipping the first version of what we call a self-service update for the Oxide Rack. So today the process for updating the Oxide system, including the control plane and everything, involves a support process where our support engineers are getting on the system through a debug interface. And that works pretty well for a lot of things, and it's very simple. but it doesn't work for a lot of customers to have an oxide person involved in the whole support. I mean in the whole upgrade operation and so we want that to be something that they can do through the API just like they can do all the rest of the infrastructure stuff.

29:32But that is kind of a big deal because that means that the control plane is driving the update which means the whole control plane is online during the update. And so people use different metaphors for this. It's replacing all the parts on the car while you're driving down the freeway or replacing all the parts in the plane while it's in the air or whatever. but it's just a lot of work to make that work. How long have you been working on that? I've been working on it for about two years. How long has Oxide been working on that? We needed an update process for our MVP. And so that started before that and finished before that because we launched before that.

30:05But that's the process that we use today in the support-based process. This is like an upgraded rewrite. Kind of, yeah. So the idea behind the first version of Update was what we called the minimum upgradable product, which was Mupdate. And so the idea here is that, you know, it's an MVP. We're shipping it really as fast as we can, but there's a lead time between when you deliver software to the factory and when you actually get it at the customer site. And that's a period in which we can continue working on the MVP. And so what we needed was for there to be enough in that first thing that we could update it to whatever software we wanted once we got to the customer site.

30:39And that became the minimum upgradable product, which is Muppdate. And that's the procedure that we have today. So the priority there was about having a robust support procedure for recovering the software on any one of our compute sleds. And that's something that we knew we needed even separately from update. And so that's why it came first was this idea that we need the ability to recover a sled. It's one of our compute sleds, no matter what state it was in, and we can use that to do our initial updates. And that's what we've been doing for the last two years. So in that sense, we've been working on it for a while.

31:09But in terms of being able to have the control plane do a more operator-friendly update, that's been about two years. Gotcha. And actually, even that, there are a bunch of building blocks involved in that, like what I've come to call dynamic reconfiguration of the system. So having the ability for any component to come and go while the system is running is kind of a prerequisite for that, but also allowed us to deliver other important things that customers would expect to be there, like the ability to remove a sled, replace a sled, add a new sled to the system. So the first year of the update project was really building this foundation that we use for these other support procedures as well.

31:45Gotcha. What exactly is an update? Describe, is it big, is it small? Yeah, no, that's a really good question because you think of the control plane as like it's the control plane. It's just like one big piece of software or something like that. But actually every single one of our, so we have in a rack, we've got 32 sleds, we've got two switches, and we've got a couple PowerShell controllers. Every one of those has a service processor, a root of trust, and the root of trust has its own software and the bootloader software. Then on all the 32 sleds, we also have a host OS, which comes in two parts for historic reasons around bootloaders and stuff.

32:20Then we have all the control plane software on top of that, including storage software, which is one per disk. What all this means is that when you update the software in an oxide system, you're updating literally hundreds of components. And we're kind of doing it one at a time. and you're also going through all these intermediate states where you're running some of the old software and some of the new software. So you're asked, like, what is an upgrade? We're replacing all of the software running on everything in the system. And it's a lot of different things. Some big deal then. Yeah, and that list I just gave doesn't even include a lot of stuff that for us gets bundled, like CPU has microcode, NICs have their own firmware.

32:52For the update system, that's simplified because it all gets part of the host OS, but there's a lot of software in the system and it's updating all of those things. But the whole idea of what we're doing is that operators don't have to think about any of that stuff. So our release process puts together a giant zip file. It's like two or three gigs of data. You download that from us. You can look at it if you want. You can validate it, whatever. But then you upload it to the API and you hit go. And then the system goes insurance for probably a couple hours up front. And then you come back and then the whole thing's updated.

33:24So the idea is that the operator is only thinking about this policy. They're not thinking about all those other things that are involved. That's nice as a zip file in several gigs though. It's a lot of software. Yeah. I mean, even the bandwidth cost on that, do you measure that? Does it matter as you grow your customer base? I mean, obviously not. I mean, because you're getting paid lots of money, but you got to worry about those things, right? In the limit. Speed to get it, accessibility. It's a good question. You know, it's not something we've been focusing on for the most part so far. It's the kind of thing where the customer is currently going to be responsible for getting that from whatever our download site is, whether that's GitHub to the Rack.

33:58So they might get that on their laptop, downloading it from GitHub or something like that. And then they'll upload it to the Rack. Yeah, that part is GitHub's problem. It's a release. All right, that makes sense. And then uploading it to the Rack is over their network. And we've kind of been assuming it wouldn't be an issue. That's a fair question whether they would consider it an issue. I'm just talking about size, the file size. I guess most people can download a couple of gigs pretty easily without it being a major problem. But some people can't. It's definitely a lot. I've been talking before.

34:24I'm like, I guess if you're... Most of their customers are not. Yeah, they're probably not having those problems. Yeah, I'm solving different problems. What about air gaps? So some people often have air gaps. Yeah, that's really important too. And that's what this model works pretty well for air gaps because you're downloading it to you, the customer, are downloading it to your laptop and then uploading it to our thing. No connection to the rack definitely doesn't care. It doesn't even know if it's connected to the internet. That's fine. It doesn't care. Right. You could imagine a nicer experience where the rack was connected to the internet and could see, oh, Oxide's just published a new thing and I'm going to download it.

34:54Maybe I only download certain parts that I need or I download it one part at a time. Maybe that helps with some of the bandwidth stuff. But actually it's not something most of our customers are interested in right now. Most of them are actually more interested in the I'm really not connected to the internet and I don't want you to. And I'm doing this because I care about my security and my privacy and my data. I definitely don't want the rack talking to the internet. So that's why we've done it the way we've done it. We talked to Cliff earlier about Hubris. So when you look at Hubris, that's the operating system.

35:19You mentioned a couple chips, every different device on there. When the update comes through, is that on top of Hubris? the API of Hubris, or is that how they compare to each other, this update and Hubris? Yeah, so Hubris is a sort of, it's an operating system. We use it in a couple different components in the service processor and the root of trust on all of these systems, on the sleds, the switches, and the PowerShell controllers. So it's one of the things that we update. It is also true that in order to update everything, we talk to the service, or in order to update much of the system, we end up talking to the service processor, which is talking to that Hubris thing.

35:55So we end up using the current version of Hubris through the service processor to be able to update the service processor itself, the root of trust, the root of trust bootloader, and the host OS as well. All the control plane stuff is on top of that and doesn't go through Hubris. Is this a novel problem that you all invented, given your architecture? That's a good question. Yeah, a lot of the details are specific to our architecture and pretty novel. It's the sort of thing that I expect cloud providers today have their own bespoke software for. And in fact, large deployments of on-prem stuff will have their own bespoke software to do a lot of this stuff.

36:30But a lot of it is also stuff that people kind of just don't update. Like how often do you update your BIOS? How often does a company running on-prem software update their BIOS? Probably not all the time. But our model involves delivering a lot of value through stuff like that. And we do need to be able to update that stuff. I remember back at one of my past jobs, we did have to go update the BIOS on like 64 systems or something like that. And you've been in the BIOS thing, right? You're like clicking through the thing and like how do you do that on 64 systems? And like at the time, this wasn't a productionized thing.

37:05But at the time, I didn't know like iTerm or whatever has this mode, which is like send all my keystrokes to all the other panes. You just taught me that right now. Someone just like opened up 64 panes and was just like enter, enter, tab, over. Oh my gosh, that would be that person. That's a 64X developer. It's a real problem, right? When you have this software, it's such a low layer that's not really designed to be interacted with by automation. Everyone that's had to do this has had to come up with their own way to do it, basically. Well, one reason why you don't update your BIOS very often is because you got to reboot your machine.

37:36And I know that ultimately your guys' goal is like no reboot update, right? That's right. That's not what you're working on now, though. That's going to be the next phase, and it's going to be a much smaller part of the problem for us. At least, you know, we expect. It's going to be easier. You never know until you're done. It's not going to take you two years. Right. So we've done a lot of the pieces involved in that. All the stuff we've been doing so far is like the orchestration. It's foundational stuff. So like our system is based on what's called the plan execute pattern, which means that before taking any action, the system generates a new intended state of the world, which we call a blueprint.

38:06And then it goes and executes that blueprint. And like all that was really important foundational work for building a system that can be operated autonomously, which is also really important for the air gap thing. Because we can go test all kinds of things that can happen with just the planner part without even worrying about the execution stuff. And then we can go test all the execution stuff given whatever plan we want without having to have gotten a system into exactly that state. And it also lets you do all kinds of things like ask the system, what are you going to do next before it does it?

38:34And why are you doing that thing? You know what I mean? So these are really important operational things that we just need to have. That's the kind of stuff that's taken the first two years. So you laid a lot of groundwork. That's right. And so now we're talking like doing what we're calling non-disruptive updates. So this is doing updates without rebooting the customer VMs. We're still rebooting the sleds. So do you move the VMs? Exactly. We're going to live migrate the VMs. And now that's a question. I don't mean to oversimplify it, but that should be just a question of policy, which is like we're flipping a bit in that blueprint that says this sled needs to be evacuated.

39:05So let's say you have an oxide rack, 16 sleds. Is that typical? Yeah, 16. All right. So I got 16 sleds. Yeah. And we need to run an update. this is in the new world when this exists and I got each sled has I don't know 30 VMs on it okay now we're doing math in our head oh he's gonna do a math problem no I'm not going to and it comes time to run update so I go put it on my thumb drive off my laptop or whatever plug it into the rack and it's going to run it's going to live migrate VMs off of a sled one at a time update that sled, reboot it, and then move some stuff back to that one probably. Something like that.

39:45Yeah, that's where actually... Distribute that load evenly across the other ones in the meantime. That's where a lot of the complexity does come in with the non-disruptive update is, first of all, how do you mechanically move these things around? It's like a bin packing problem. But then there's also how do we make sure that we have the capacity to do that? Right? If we're going to start doing it up, if you've totally filled every sled... If you're running your thing at max... Right, there's no place to put it. You have nowhere to move them. Right. Buy another rack. Right. Well, that's the thing.

40:13At scale, people actually don't care about this problem. Because keeping a couple of sleds capacity free when you've got 100 racks is a very small fraction of your cost. And it actually makes sense for a lot of reasons. It also allows you to sustain failures and put that stuff over there. But when you've only got one rack, that might be more of a problem. So then there's the question of how do you create an experience for the operator that communicates clearly what the tradeoffs are, but also gets this input from them, which is like, what do you want to happen? Do you want me, the rack, to prevent you from using all your capacity so that you can update it?

40:42Or do you want to have the possibility that you go do an update and we just say, sorry, we're paused right now until you can tell us where to put stuff. Reboot all these VMs or whatever you want to say. That reminds me or that makes me think of like failed updates, you know, in the self-service world when this version's out, not the, what do you call it? Undisrupted? Undisrupted. Yeah, uninterrupted. With the current iteration you're working on now, can you guarantee that an update will finish? That's not what I thought you were going to say. Finish? No, because there's things outside of our control.

41:19Like, for example, one of the biggest challenges in self-service update, because the control plane is running, we have these intermediate states I mentioned where you've got new version of software talking to old version of software. And how do you avoid that becoming a problem on our ability to change our own software because you have to do backwards compatibility like forever, right? And one of the ways we've addressed that is to say that there will be an order to the updates. So we will always update, for example, the host OS before we update the control plane that talks to it because the reverse never happens right now.

41:49And so that's fine. But that means that if you're doing an update and one of the sleds is like out to lunch and we can't talk to it, we don't know if we've been able to update it. We can't actually keep going and update the rest of the control plane. You've got to wait for that sled. Yeah, so we've got to tell the operator, look, you either need to what's called expunge this leg, which means to remove it from the control plane and we'll pretend like it's just caught fire, like it's failed, we've moved everything else elsewhere, or you've got to figure out what's wrong with it and bring it back, and that would be a support call, probably a support call.

42:15And then unless they go, they just unplugged it and they just plug it back in or whatever. Those sorts of things are always outside of our control. Yeah. Now, what did you think I was going to say? I thought you were going to say an update that explodes, that you start doing the update. That was my next question. And the control plane is now down, and what do you do? And like that's not that's my nightmare. That's been my fear for the last couple of years. That's why it's taken two years and I'm not done yet. But seriously, that is why we spent so much time on this like care. Having the automation take these careful steps where every one of these steps we know is safe.

42:46We've taken, you know, as an example, we've got a cockroach DB cluster. That's like storing all the control plane database data. We've got five nodes. We definitely don't want to bounce a sled that's hosting a cockroach node while that cockroach cluster is already unhealthy, right? That's just like a thing we want to make sure we never do because that increases the risk that we actually lose quorum on the cockroach cluster and it's dead and we're in trouble. So we have all these kinds of safeties built into the automation, all this testing, this whole pattern and all this stuff. So that's like one angle.

43:15Obviously, testing is another example, angle, but that's kind of a given. Sure. But it's a hard problem, really, because part of what we do, part of what is involved in an upgrade is making backwards incompatible changes to data formats, like database schemas and things like that. And once you've done that, the old software can't read the new thing. So rollback is really not possible. So what a lot of software does is it'll have a point of no return, whether some call it a finalizer or a deferred update or something like that, where you basically get the whole thing kind of working before you've committed in that way.

43:50And then you ask the operator, like, does everything seem to be okay? And then they hit the button and it's like, okay, fine, go. But even then, there's like, there's still risk there because whatever it is you're activating by taking that last step hasn't been tested before that. And there's kind of no way to get around that. And that's kind of a future problem for us right now, but it's something we're going to have to deal with. So it is your nightmare. Has it ever been your nightmare? Meaning it's happened? Like in my career? Yeah. Well, like specifically for oxide and updates. No, no. Any updates gone wrong?

44:23No, but we haven't started doing that in production yet. So there's still time for how to do it in production now We're using that the the process we used when yeah The manual thing when we so the way the manual process that I was talking about the very beginning works Basically shuts down the whole control plane replaces all the software and brings back up again. How long is that process? It's actually not that let's be shorter. I feel like that's safer. Why? I mean, I get the whole reason down time man can't have it. It's downtime and it's the self-service aspect Although you could imagine a self-service version that looked more like that.

44:54But then the thing is, if it's self-service, you're talking to an API, what's running that API while the thing is down? There's nothing, right? Yeah. So that's why you got to do what we've done. It's got to be done. Yeah. So downtime is required now. Future is, you said non-destructive? Non-disruptive. Non-disruptive. meaning the VMs get migrated around versus shutdown, but you still reboot sleds. That's right. You're still rebooting controllers and control planes and stuff like that. Yep. But that's not visible to them because we have enough redundancy. Is there a world in which that isn't even required?

45:33Which part? What if you didn't have to reboot anything? Oh. I think that's pretty dicey. It's definitely a thing that people have done. Yeah, hot swap? Yeah, there's types of updates or patches to a running Linux kernel where you write the new one over here and then you jump over there. It's just a pointer. Right, it is. It's all software, right? That's right. Who says we have to reboot? Yeah, that is definitely harder to do with stuff like the SP and the ROT. Part of it is that the ROT's job is to attest to the software that's currently running. Yeah, if you're going to change it out from underneath it.

46:12Yeah, you need to figure out what that means. And then there's also this risk that you're now in a different state than you would have been in if you had actually bounced that thing. So have you created a time bomb for yourself where if that thing loses power and actually does power back on, is it going to do the same thing that it was doing? That's the thing I always... Like bifurcated code paths where you're like, this is the thing we do sometimes and this is the thing we do other times is totally the kind of thing that results in something failing at runtime. Catastrophic failure. Yeah. All right, fine, bad idea.

46:41Yeah. The thing you asked about like my nightmare, right? The upgrade just like explodes and we're toast. Another thing that we've done there is try to create a lot of guardrails around the types of changes that we can make to the software so that we know if we're making a change that's going to break things. So, for example, if you're changing the database schema, we know that you're changing that and we've operationalized that one, so that's kind of fine. If you're changing like an internal API, we make sure that you're doing it in a way where none of those intermediate states will expose us to a situation where those components don't speak a common version.

47:15And like that's something I imagine other organizations do do. I haven't actually really seen that before, but I think it's really important. So because that's the way that I've seen this fail in the past is like someone goes and makes a change to the API. They're like, I'm not changing upgrade. So like they test everything, everything seems to work. Right. You go deploy it and like it blows up in the middle. It's like we've tested the endpoint, we've tested the beginning point, and we just got unlucky in one of these intermediate states that wasn't tested. And so we've tried to identify the kinds of changes that would cause those problems and then detect those at CI time for us and at build time if we can.

47:48That's cool. Any newer novel testing strategies that you've had to come up with as far as this, I don't know, fuzzing or deterministic testing or anything that's... That's probably the biggest one. The other thing I would point to is that sort of distinction between the plan and execute stuff. We haven't actually gotten into this kind of thing. But one of the things we want to do with that is like property-based testing on the planner where you're basically like sending all kinds of different inputs at it and putting constraints on what kinds of outputs can happen and see, you know, make sure it never does anything crazy.

48:20Yeah. We were supposed to give a talk today. Yep. It got rescheduled. That's true. We're not going to be here for it. Right. So you have to spill the beans. Yeah. So let's see. What was I going to talk about? So I've been doing this for a couple of years. Update on update? We have a surprising number of new faces. So part of that talk is literally the stuff we were just talking about. Like, what is update? What is it? Well, we've got a couple of hundred components that we've got to replace and pick your metaphor or whatever. So part of it is just like laying that out and like what we do today, why the stuff we've been talking about, why this is a problem for customers, what we're doing, current status of that, which is like we're planning to ship the self-service part very soon now, and then non-disruptive is coming after that.

49:02Then the rest of it was probably, I don't know if that's really interesting to a broader audience, but it's kind of reflections on what it's been like to run a project for such a long term. And I don't know, maybe it is more generally interesting, but like I have a lot of fears about update at runtime, but my big fears about update as a project was that it would feel perpetually a year away. And we would make decisions day to day and week to week that ensured that it continued to be a year away. Because when something's like a month away and somebody asked you to do something else, You're like, sorry, I got to do this thing.

49:35Like, we're shipping it in a month. But when something's a year away, it's very easy to be like, well, here's a really important problem over here. And, like, it's hard to know what the next thing is to do on update. That's been really a challenge the whole time is, like, what's the next step? There's so many steps and there's so many circular dependencies in those steps that you're like, well, I got this other important problem over here. Maybe I'll just, like, kind of solve that. And that's fine. Sometimes that's the right call. But if you make that call. Over and over again. Even 20 % of the time.

50:01Right. Exactly. You just never get there. Your timeline stretches. How close is it? How close? You said very soon. Didn't you hear him? Yeah. Update. I'm just kidding with you. Yeah, I mean. Within the next month or? That's the plan.

50:15That's not a year away though. No, no, no, no, no. This was two years ago I was worried about. Two years ago I think I was like perpetually a year or two away. I saw the roadmap this morning. Yeah. I saw last year's roadmap. It's not a roadmap. Did you gift them? It's important things. You know, where bodies are buried? One of these was update. Yeah. And then the priority this year is update. I thought it was top left, which was first. It's ongoing. Accurate. But almost ready. Yep. Almost ready. That's cool. Why you? So we had a side conversation. He's been with Brian almost his whole career. Son to Oracle, the acquisition, and then Joyent.

50:51Okay. And then what makes you uniquely positioned for this task, this quest? I don't know if I'm uniquely positioned, but right after we shipped the MVP, this was one of my big worries about the products. And I was like, this is something I think I have a lot of experience with in terms of building distributed systems and reliable automation and things like that. And so I thought this was a good opportunity for me to swing in and try to create the vision that I want us to get to and then be able to execute that. So I was interested and I asked and that's how a lot of stuff works around here. What has it been like casting the vision for it and getting feedback from the team?

51:29because I'm thinking culture, I'm thinking the process of getting feedback on any idea because you're laying out all this groundwork and you're doing all the work to kind of get to a direction. What is it like to put that idea out there, that vision out there and get that feedback and start moving on it? It's great. It's great. I don't know if you all talked about the RFD process, but... The RFD process, yeah. Yeah, so when I started on this, the first step was writing an RFD. I think it's 418, if you want to find that. And it's basically... Is this public, the RFDs? some of them are and some of them aren't.

52:01I don't know if that one is. It probably could be if we want it to be. There's nothing particularly sensitive in it. But it's kind of laying out where we are. Lay of the land. This is where we are. These are the problems. Here's where we're trying to go. And like it was very specific in some ways. It was like this idea of plan execute pattern and the automation has to be safe and all this stuff. But it was also very like we have a lot of stuff to do and I don't know what all the pieces are yet. So that was the first step is getting everyone aligned on the vision. And that RFD itself was a team effort.

52:28I drafted this first version, but people are looking at that. And I think broadly, there were no surprises there. Everyone was like, yeah, this all makes sense. And then that process just keeps happening. You get more and more specific designs and say, okay, let's get some feedback on this. And I do enjoy that part of it. And I enjoy the collaboration. And it goes pretty well. It's not the sort of environment where you're worried about what so-and-so is going to think about this. And is someone going to be unproductive about it or something? We know that Rust is a foundational language here, obviously.

53:01Yeah. We talked about this being somewhat of a novel problem. How is Rust uniquely positioned to help solve this problem? What about Rust makes this problem easier than another language that you may choose to do this with? Yeah, so the big thing for me about Rust that I really love and I think that's been huge for Oxide is its ability to help us ensure things, especially at build time, that need to be true of the system. And so that sounds really vague. But what I mean is you can catch so many problems early. And everyone talks about the obvious ones. We talked about this earlier, but everyone talks about the obvious ones.

53:38Like it's the borrow checker will help you find memory safety problems. But it also allows you to create abstractions for the rest of the team that can't be misused, right? So you can say, I'm creating this thing. like maybe you represent it with an object and you say, you can't do these two operations concurrently on it. Well, that's the thing that we can enforce in the type system. And like you literally just can't compile the code that would do that. That's awesome. And it sounds so low level, but that's what a lot, like it's extending that same idea and applying those same tools to do it that allows us to say, if you try to evolve the API in a backwards incompatible way that won't work at runtime, at upgrade time, you might get, depending on how you do it, you might get a build failure.

54:23You'll at least get a CI failure. And so I don't know if we have a minute to talk about it. I mean, so one of the first things that I built here with Adam Leventhal is something called Dropshot, which basically lets you write an HTTP server and then generate an open API spec from the code. And then we feed that into something called Progenitor, which generates the clients for it. But that alone means that if you make an incompatible change to an API, even before we'd done any of the versioning stuff, your client fails to compile now, which is hugely valuable. That's true not just because it doesn't have the operation in it, but you passed in a Noon with three variants and it now has four variants and you need to accommodate for that fourth one or something like that.

55:05I call them enums. Yeah. So anyway, we ended up extending that. So now that's how we do this versioning stuff is we have a bunch of these open API specs that are the ones that this thing supports. And then we know if you've changed it because we know if it generates a different thing, then you've changed it incompatibly. So Rust has facilitated all this stuff. And sorry, the last thing I want to say about that with the Dropshot thing is like, you've got your rich structured types in Rust and you're like all happy because I'm in Rust land and I've got my strong types. That carries all the way to the client because of the way it goes.

55:37Like Dropshot just takes those types and puts them in the open API spec and then the client generates faithful things on the other side. So you can basically get that strictness all the way through there. We gush a little bit more on this then. Speak to your confidence in the code you write because it's Rust. I'll just easily shot you that one. Yeah, I mean, I'm reluctant to make any bold claims about the correctness of the code I write lest I immediately walk back in there and there's some horrible thing happening. The fan spin up, that's why. But I mean, I'll say this. There's a lot of changes I've been able to make where I'll go work on the code for like four hours.

56:12and when it compiles, I know it's already correct. I know I haven't broken anything. And it's not as simple, you know, I know there's the cartoonish version that's like if it compiles, it works. And I'm not talking about that. I'm talking about something that's like, it's either a refactor, this happens a lot with refactors, or I'm building a new thing in terms of these things that already exist and I'm plugging into the middle of it. And it's like, there's no way for this to be wrong at this point because like it fits neatly into the narrow interfaces on both sides of it. It's correct. And that is huge.

56:45It's so huge. No nightmares. My previous experience was in Node.js and it was the complete opposite of this. And part of the reason I love Rust is by the time we got to the end of the road at my last job when we were using Node.js everywhere, every single JavaScript function we had started with like 30 assertions about the types of all of its arguments. And I'm like, why are we doing this? The computer can do this. It's what the compiler is for. And so I have so much more confident in those things now. And that gets back to why I think Rust is so valuable for this problem space is that these things are so complicated.

57:18By allowing us to encode all these constraints into build time constraints, that allows us to evolve the software so much faster and with so much more confidence. Someone can come in here and make a really big, complicated change. And you're not wondering, like, well, what about, I wonder if they missed some call or something like that. It's like, no, you haven't. You've covered every single case. And it's huge. It's huge. Right on, man. What else? Anything? Tell us about Oxcon. Oxcon. Oxcon is awesome. So this has been something we've been doing for a while. And this is something we, it's very similar to something we did at Joyant, these sort of company-wide meetups, engineering-wide meetups.

57:57You know, for a remote company, it works really well to be remote basically all the time, but it is also so valuable to have that time in person together to get to, there's so many conversations that don't happen if you haven't scheduled a meet for it. And we have other ways of trying to have those conversations anyway, but it's just really nice to have the FaceTime with people. And also, like the small talk and like you're going to get dinner and you're just like talking to people about whatever it is. You learn more about them. It's a really good time. So there's a mix of the structured time that we have.

58:28You know, Brian and Steve talking about all the exciting stuff over the last year. It's inspiring for people. It gets everyone ginned up to talk about everything that we got to go do. and then there are other good company-wide sessions talking about important projects and stuff like that. But then there's also all this breakout time that I started by talking about, which as with any conference, the hallway track is almost, it's at least as important as the rest. It is the best. Yeah, right? So that's huge. Not really. Hallway track only. Keynotes only. And then hallway track. That's our move. This is so much more, there's a lot, it's getting more like mature and professionalized.

59:05We never had a stage before. We never had like pro AV before. So this whole week is a little bit of a dream. It's like Occon already feels different. And then we've got the stage there. And then it like rained in the Bay Area in September this morning. And I was like, am I dreaming? What's going on here? Yeah. But it's a good time. Thanks, Dave. Thank you. Yeah, thanks. Appreciate it.

59:26Well, friends, last up is Ben Leonard in charge of all things design for Ocside. Here we go. One of the things that stands out about Oxide is its design. I think you can have a good company, successful company, but every company that's successful is set apart by its design. Can you talk about how you came to be here at Oxide? Yeah. And the design story behind the work. Yeah. So I was working at a branding agency for Pentagram. Oxide was my last branding project. I was planning on leaving to work as a freelancer. I'd been at Pentagram for four or five years. I was ready to move on. And it just so happened that Oxide were looking for a designer just as I was leaving.

1:00:10And it felt like a unique opportunity to... Because what would happen is you'd work on these brands and then you'd throw it over the fence. And then you'd check in on it two years later. Like, what have they done to my baby? They destroyed it. My boy, my boy. and so it was a unique opportunity to continue it on and I'm kind of I was trying to you about this before I'm like relentless that I like to fiddle with things and I can't work it on it yeah I kind of get bored of it as well so I just want to I just want to tweak something so we I mean recently I mean like we made a modification to the to the logo that most people wouldn't even notice it's slightly thicker but it's something that's bugging me this entire time yeah yeah yeah When Steve was up here, I was like, Steve's shirt has the thinner logo.

1:00:57And on stage here, it's much thicker. I noticed that. I saw it. Right away. I saw it. I noticed turning. You need to change that. Oh, yeah. Yeah, yeah, yeah. For sure. So I'm always kind of, yeah, always filling with stuff. But then the beauty of OXIDE is I get to work across everything. So I'm working across most generally, most of what I do is working on the product. So in my case, it's the web console. So you work on the design system, the UI. I'm also working on the brand and the marketing and the sales assets and all this stuff and industrial design. The beauty being is that those all inform each other.

1:01:30What you tend to happen is you have your product team and your creative team, and they're very distinct. So one, this is a bit in the weeds, but we have the same UI design system that drives every piece of design. So we have the same colors, we have the same UI elements. on both the website and the web console. They're kind of continuously, continuously informing each other. And there's, I mean, it's difficult because I think there's been a, like a shrinking of creativity in tech design. That is, branding stopped being, what it was is that the product itself, so the proliferation of SaaS, the product itself became the design.

1:02:18So the UI language of the product became the design. So linear is a great example of this, which is the design of the product and the design of the creative are one and the same. What that can mean is you have the world of design within product is much smaller. And so what that can mean is you have everyone's always complaining that all tech looks the same now. And that's probably because product is much smaller. And if your brand looks like your product, then your world is much smaller. So I'm kind of, I'm always figuring out how wide I can go with the branding and keeping it still like it comes from the same world as everything else.

1:02:58And yeah, it's a, there's, yeah, the industrial design, the creative, the product, there's so much to work on. This rack behind us, would this rack behind us look like that at all if you weren't here? um it'd look a bit like that i think it'd be same shape yeah if i wasn't here it'd be really it'd be it'd be squat it'd be other size no i think less green probably it might be less green um already more green we went back to uh i worked with so we we consulted a little bit with um my my old agency pentagram um industrial designers there and they kind of we worked together on that little bit. One thing that I'd learned from working on a hardware product previously is, the first version was, it was a small run that they distributed to people, right?

1:03:49And so they produced this thing, which is initially really magical. And then as soon as they mass produce it, they think, okay, we're going to have to make this cheaper, easier, more practical to make. And then the next version looked really, really bad. Because essentially, they were just trying recreate what they've done and and okay we can't do this complicated tile system so we'll just print it directly on on it was a pc pc e code right um and and what and what it was is it was compromised in the worst way so i think like going into it from the beginning thinking like what's the what are our limitations like we're gonna make we're gonna make thousands of these like how from the beginning i don't want to i don't want to do things that just exist in this kind of first this initial run like and so figuring out what the um what those compromises are but i'm working working with heart like industrial design working with hardware hardware is hard i haven't done very much of it um but uh yeah you you're as soon as you're touching materiality you're you're dealing with things are different yeah i mean we were speaking about like color color is the bane of my life like we have we have the we have painted dry bays we have powder coated metal So we have like plastic pieces and color matching the green, I think.

1:05:02Yes, that's probably got to be your thing, right? Yeah. Is that a little off sometimes? Oh, it's a little off all the time. It's a lot off all the time. And some of it's outside your control, so you have to think about how to handle that. So what you do is you avoid putting elements next to each other that might be different. Yes. But like if there's a little bit of separation, then visually you can get away with like a little bit of difference between those things. um we had so one one i mean one example is originally the rack the was like it was a black but it was a slightly bluey black um like bluey as in the cartoon blue exactly yeah it was like it was like a bluey black disregard it was it was it was a cool black cool as in like color temperature not like cool black i thought you liked it yeah yeah um but the my my thought was as soon as we start integrating multiple components, then I'm trying to make sure I'm using the exact right black.

1:06:02Oh, possible. Yeah. That's hard. Yeah. Although I don't know how much of it is me trying to avoid upsetting myself. Like these things that no one else notices. They're like, whatever. Ben's upset. I don't know. This is like trauma-driven design. Right. Okay. Yeah, you guys have TDD. I have the same. Do you get into the PCB design then too? Like how much, or is it simply this, I don't want it simple as a flyway to do, but do you get to step into that world where like, how is this? Because I think about Apple. Apple has done a great job of like branding everything. Look at their CPU, right? Their latest M1 chips are the latest M series chips.

1:06:43It's a design to everything they do. Do you get into that as well? I mean, Apple's a good example because they're stunning inside as well. A little bit. Occasionally I dip my toe in and I get shouted to leave. So the PCBs are currently there. The PCBs are green. And I think once I came into a channel and I was saying, Hey guys, can we make the PCBs black? I'm thinking, okay, you changed the color of PCB. You just order black ones instead of green. I was told that's not the case. I think it does have impact to like thermo and stuff like that. What did they tell you? They told me some information, none of which I retain.

1:07:22You're like, sorry I asked. Maybe they said it, maybe they weren't telling the truth, maybe they were just saying it because they were like, yeah, yeah, yeah. Light looks cool, I've seen light PCBs. But there are issues with, yeah, there are manufacturing ramifications which mean that it's a bit more complicated than that. I think with the rack you have to take, there's a balance. We can't go full Apple and just invest so heavily in details which cost money for a little benefit. But what we can do is make sure that something which is an item which is underappreciated. I like taking boring objects and boring designs and boring, boring things and making them better.

1:08:07I think it's easy to come into, it's easy to work with Nike and make and do some beautiful design for something that's already like really compelling. But I think it's much better to take something which is for which design does not usually kind of feature and try and elevate that. So Offsite is big on the values. You know, you go to the website, there's values. I'm sure you know all the values. They inform all the decisions, how you act. do you have like a design language or value system or something that that drives your design for oxide that's separate or maybe even congruent with that there are i think there are there are there are ideas so uh i think my approach to branding is you have one you have one idea which should then filter through into everything and that's how you make holistic thing it's not necessarily that this looks like this it's that it comes from the same place yeah so with oxide oxide is an old idea like broad new right so it's this it's like this old crazy idea that you own you own your own hardware um and so you see that in the design language you see the ascii i mean the green is like this kind of terminal green right like it looks it it it has like this kind of retro edge to it and so the design language is like old meets new and yeah so i think that's like nostalgia is a key part of it.

1:09:30I mean, you see over there, we have those, the logos, nostalgia is a key thing, but you don't want to, you don't want to be so in a style. You don't want to lean so much into nostalgia as kitsch. So yeah, I think that sort of referential, referential thing to kind of old computing is a, is that maybe that's why it speaks so well to us because we're all about that. Yeah. Where the old meets the new is interesting. Precisely. What gets you excited about what you do here? like what makes you be like hell yeah i like you say hell yeah i don't know the british equivalent of hell yeah is um i don't i don't think with that um yeah i don't think with that with yeah we're not that emotive yeah um we'll get you properly excited i now you're talking his language i'm properly excited um no i uh i i just i i love to do a bit of everything and it's the variety i think that really drives me i think it's why i mean i've been here more than four years now i think that's what variety is what keeps it interesting and like upside the company where you get to do that you have hardware you have software and obviously and you you and everything kind of in between and yeah yeah so yeah i get to that's the thing that keeps me keeps me keeps me interested is the young i would expect the the opposite because at an agency you think that's where the variety comes because you're on to the next project, like new company, new design language, new brand.

1:10:56Yeah. Whereas you decided to settle down with one brand and do that for years and years, I would expect it to be less variety, but you found that it goes wide. I mean, I think certainly working at an agency, I think you're always going to, it's always going to be much more varied than working a startup. But I think for me, it's, um, there's this interesting tension between what's staying the same and what's different. And so there are things in the Oxide brand which have retained logo more or less, the colors. But I think there are things that are kind of constantly changing. And obviously, like, I think Oxide as a company is changing.

1:11:31So design needs are changing. Sure. As an example, I think we are growing and sort of like marketing and sales needs are growing. And so previously, there was very little collateral outside of the rack, outside of the web console and the website. um that was it so you had that kind of small world but then when you're kind of entering the world of sales and marketing then that kind of opens the door to a bunch more design stuff that thing in the door when we came in was uh as a card with like a chip on it did you design yes yeah yeah it does yeah it has a little yeah yeah it's got two sides you're afraid one that i can actually use the coin for the coin yeah one one to sell on yeah yeah yeah exactly that yes yeah that's fun stuff right yeah i mean that's what i think those are the palette cleanses so the the the big pieces of merch tend to they tend to get made just as i finish something that's like that's like drained my creative energy website is the big one because i mean i'm i'm usually coding the thing too so i've i've designed it i've coded it's took it's took so long that i'm ready for something else so we have i don't know if you saw those the little kind of rack stickers um those i designed right after I'd done like a run of the website and so those things are a nice little uh they're a nice little cherry on top they're a way to kind of refill my like creative cup a little bit are you excited about the growth I imagine with new footprint we're doing video we talked to a little fourth wall breaking here behind the scenes we talked to Tor about being able to put motion graphics into place and collaborating on that is that exciting to you to see the different areas you can do.

1:13:11Yeah, for sure. I think like there are, I have, I have a list as long, as long as my arm of things that like I will eventually get to like motion is one of those things. There are things that I want to do, but you don't necessarily have the reason to do. Yeah. So, um, I think you can, um, I was chatting about this before, which is like the size of the gamma of like creativity of things that you can do. Product design is much smaller because it needs to be functional. Um, people using it every day. Uh, there's also like, there's like guys, some of the way that people use products and you need to be, you need to be designed in a way what's expected.

1:13:42Yeah. And websites are a little bit larger and then the kind of creative space is even larger than that. And then you just think about what are these like one-off excuses to do something different. So like product launches, all those things that you can, you can get a bit more experimental around. We love the work you do. Yeah. It's great work. It's beautiful work. Thank you very much. I think the company is awesome. I think the design really, in my opinion, is like it's the glue. You can have a great product, you can have a great software, but it's the final piece that says this is just super awesome.

1:14:20Basically, the design to me is what sets it apart. Yeah, I mean the design is what's like says you care about everything. Yes, it shows intention, it shows trust, it shows all these things. Yeah. It's just really good work. so getting to go to Oxide's headquarters for this conference Oxcon was years I would say in the making didn't happen overnight and over the years we become closer and closer friends with Brian and Steve and the rest of the team there behind Oxide and just getting a chance to celebrate with them this moment this series B moment this massive new order moment this crossing the chasm moment I know for Jared and I was quite awesome so glad to share that with you glad to get this on the pod glad to peel back the layers with cliff and with dave and with ben these different parts of oxide these integral important and different parts to truly see what's on the inside of oxide now the good news is i guess even better news really is i think there's more to come they light working with us the elect will be done so far for them and i think you're going to as well we're producing a 10 minute inside oxide documentary for the youtube channel So it's kind of cool.

1:15:32In the works, in the edit booth right now, in motion as we speak. But man, I've seen previews. I've seen the teaser. It's epic. A massive thank you to our friends who sponsored this podcast, our friends over at CodeRabbit, our friends at Depot, and of course, our friends and our partners over at Fly. Check them all out. They love us. You should love them, if you will. We appreciate that. Of course, the beats are by our good friend and longtime friend, brother, Brake Master Cylinder. Love those beats. Okay, that's it. this show's done. We'll see you soon.