The state of homelab tech (2026) (Friends)

28:01Maybe you're doing some joins, maybe you've got multiple databases and the final transform is in, is in the gold layer where you're taking maybe two or three different databases or two different data sources and you're merging them in production. That's cool. Yeah. Yeah. That's cool stuff. Let's go to, let's go back to Chris from crosstalk and unleashing let's just say claude claude opus 45 on your uh udm pro or whatever if you're tim techno tim maybe you've got the latest greatest i don't you know tim i'm i'm a chocolate dot over here i gotta buy my own things and i know you probably buy your own things but you get gifted a lot of stuff i don't mean that negatively no no yeah i mean you get to play with the fun stuff i'm envious so here i am with my udm pro that's not even the special edition It's just the one that's not special.

28:52And so that's what I'm using. Yeah. Yeah. It's yeah. I wouldn't worry about it. Like at the end of the day, like their software continues to evolve. And so you're getting the latest and greatest everything, even though your hardware might not be up to stuff. That's the cool thing about unifying. Yours does like light dances and stuff. You got all your RGB stuff, bro. I mean, like I want mine when I put on my beats and I take my dance break. I want it to dance and, you know, do a light show for me. I just don't have that capability like you do. Oh, yeah. No, no. Or you can play Snake. Did you see that video of someone playing Snake on there?

29:28Yeah, it's pretty wild. On their display. That's right. So this world, something that happened recently with me, one of our neighbor friends came by, one of my son's friends came by, and he brought a Switch, his Switch 2, as a matter of fact, after Christmas. It was one of his presents. and uh like anybody who's inviting somebody with a device into their home what do you think i said i said you got to be on my guest network well for some reason they just couldn't get on like the authentication happened to the wi-fi network you know all things checked out as good couldn't get dns and so i thought it was my newly homegrown rust project which is called dns hole so i rebuilt piehole in rust Tim if you didn't know this it's not available yet dnshole.dev in the future very soon I'm waiting for one or two more things to happen before I can do that but right now even as we speak my DNS is being resolved by my own DNS server that has fully replaced what piehole is I think you'll love it when I can release it matter of fact I'll share it with you soon if we can all that to say is that he couldn't get on via DNS and I'm like gosh dns hole maybe you messed up here it was not dns hole okay it was dns hole was perfect you know what it was what's that it was vlans man it was my vlan rules okay so of course i popped out clod and i'm like what's going on here because i couldn't figure it out on my own and i'm like gosh why didn't you just pull out clod and like let it just log into your unify and just check out some things and so through investigation it turns out i had some jacked up VLAN rules and it while it was in there was like you've done this all wrong you know this is great work here Adam but like you got old rules you got these rules that conflict you got this one rule that does nothing you got this whole set of rules it doesn't make any sense can I fix this for you please sure Claude please please help me out five ten minutes later beautiful VLAN scenario again all the world's great he's on the internet they're playing and having fun they're playing Mario Kart and life is good.

31:29So, I mean, like that's the world we live in, Tim. You know, I can't even get my VLANs right, but Claude can. That is awesome. So I haven't used Claude in that way. Like I, to be honest, I haven't used Claude all that much. I, you know, I, I use a co-pilot, you know, with, with models, you name the models. But no, that's interesting. So did it do it through the CLI? Yes. And the API. yeah gotcha well it knows the ip address it has my auth i've got an ssh key so i'll ssh into my into my udm pro gotcha so it's me it's me so it logged in it could dude listen okay hold your hold your seat it reads the mongo database directly it will update the mongo database directly i know that's it's not cool in production don't read and write the database but I've done it before and so I was confident and then it will trigger whatever it does to like let the UI catch up essentially like the cache layer that's in the UDM Pro or whatever just because you change the database doesn't mean that the reads come back quickly you got to sort of recache the cache kind of stuff and oh yeah man it's so cool it will log in to the UDM Pro via SSH as if it's you or in the case of Chris which I'm sure he did or was thinking about doing is you can use your own, you can use the API, the Unify API, or you can just log right into it and just SSH around, just CD directory and, you know, like you're on a system, like you're a sysadmin, no different.

33:05I think that's such a wild world. I think that's what's making HomeLab more special to me. Proxmox has got a little more fun. Tim, if you, I'm going to have to show you some things, okay. I have a CLI. I have a CLI I built called PXM stands for Proxmox and a one liner in a one liner, Tim, I can have a brand new Ubuntu machine running. I can specify the IP address. I can specify the CPU, the RAM and the disc. It already has my SSH key. And literally in less than 10 seconds, it's reporting the IP address back to me. Yeah. Via the CLI. Yeah. Yeah. And moments later, that same, I can do PXM info and then whatever the VM ID is.

33:54So PXM info 104, for example. And it reports back to me, SSH user is Ubuntu at whatever IP address, all that good stuff. Whatever the details are of that machine. And moments later, my agents can be building on brand new infrastructure. Yeah. Isn't that cool? Yeah, that is awesome. Yeah, no, it is awesome. This is exactly what I'm talking about. This is exactly what I'm talking about. It's like, you know, AI and agents in general are just letting people, you know, get these ideas out of their head and tinker way more and go way deeper than they used to before. And, you know, this is this goes this reminds me of, you know, when I went from IT to to a software developer, you know, I went from using other people's tools to using my own tools.

34:40And for me, that was like a light bulb. I was like, I don't need the UI anymore. Give me an API or even a CLI and I can figure it out. And for me, that was just, you know, like a light bulb went off. And it was just like this moment where I was like, I felt like so much freedom, you know, to be able to build software that I wanted. And so now it's just so awesome to see like other people being able to do that, you know, take that step from using other people's stuff to using my own stuff. And so now I feel like like you, you know, I mean, would you have ever written that thing out in Proxmox, you know, five years ago?

35:18Maybe, but it would have taken a long time. It would take me way too long. I wouldn't have the time. I probably would have. It's just too daunting of a task to do. Because it's really time. It's not necessarily ability. And I suppose it's probably both time and ability. But yeah, I would have just never tackled it because it had just been too hard of a mountain to climb, really. Because, I mean, even with the augmented AI tools, it was still hard. I mean, it didn't get easier. It got easier to move faster and to get past the hurdles. But gosh, I had to solve so many problems and figure out so many ways to deal with how do you store the image on Proxmos.

35:53Well, that's kind of obvious to most people, but like getting through the whole life cycle. And that was one of the first things I built with Claude. So I've learned a ton since then. So I want to rebuild it. I want to go from, because now I know what the tool should do. And before I was trying to make this, I don't know what I was trying to make. I was just trying to explore really. And now I know exactly what I wanted to do and what I don't really care that it does that I just don't need. And so I just wouldn't waste my time on that parts of it because I was trying to make this, I guess. I didn't want to have to log into my process machine every single time and navigate the web UI and click all the things.

36:31And it's not that it's a bad UI. It's just that that's just the that's not the way anymore.

36:57This is the year we almost break the database. Let me explain. Where do agents actually store their stuff? They've got vectors, relational data, conversational history, embeddings, and they're hammering the database at speeds that humans just never have done before. And most teams are duct taping together a Postgres instance, a vector database, maybe elastic search for search it's a mess our friends at tagger data looked at this and said what if the database just understood agents that's agentic postgres it's postgres built specifically for ai agents and it combines three things that usually require three separate systems native model context protocol servers mcp hybrid search and zero copy forks the mcp integration is the clever bit your agents can actually talk directly to the database they can query data introspect schemas execute sql without you writing fragile glue code the database essentially becomes a tool your agent can wield safely then there's hybrid search tiger data merges vector similarity search with good old keyword search into a sql query no separate vector database no elastic search cluster, semantic and keyword search in one transaction, one engine.

38:19Okay, my favorite feature, the forks. Agents can spawn sub-second zero copy database clones for isolated testing. This is not a database they can destroy. It's a fork. It's a copy off of your main production database if you so choose. We're talking a one terabyte database, fort in under one second. Your agent can run destructive experiments in a sandbox without touching production and you only pay for the data that actually changes. That's how copy on write works. All your agent data, vectors, relational tables, time series metrics, conversational history, lives in one queryable engine. It's the elegant simplification that makes you wonder why we've been doing it the hard way for so long.

39:04So if you're building with AI agents and you're tired of managing a zoo of data systems, check out our friends at TigerData at TigerData.com. They've got a free trial and a CLI with an MCP server you can download to start experimenting right now. Again, TigerData.com.

39:27I wanted to be able to do a CLI version of it. I wanted to get JSON back and feed it to my agent. And now that's all possible, really. So have you played with or heard of this latest thing, which is called Ralph Wiggum? Ralph Wiggum. No, that name sounds familiar, though. From The Simpsons. OK, yeah. Gosh, what is why is it called Ralph Wiggum? I forget. I think it's because they just keep trying despite setbacks is how I if I can paraphrase Ralph Wiggum. is keeping the leap going, these spite setbacks. And so I believe it was, yeah, I don't have it here. I was going to try to figure out who actually created it.

40:14I think his name was, I don't know, I can't remember, but it was somebody who discovered this loop essentially. So you essentially keep feeding back the loop of the input output that you would normally do with your own typical cloud scenario, which is you entering the prompt, it doing some thing and returning some sort of response back to you and doing work in between. Well, they have found this way to create this Ralph Wiggum loop so that you can essentially define a pretty clear instruction set. You might call it a spec or a spec, but they actually just call it prompt.md. And so in this prompt.md, which you would feed into Ralph, can do a loop.

40:52It could be a small loop like build this one part of the feature end-to-end, and you just go until it's done. Well, the reason why I'm telling you this is because I feel like now that I know what it could do and what it should do, I would want to, and if hardware was more available, I would be more inclined to do this. But I would build a test subject hardware machine that is Proxmox. and then I would just set loose this. Now that I have a pretty clear vision, I would set loose this thing on that machine and just have it build this Proxmox redo, I suppose, potentially, because I'm just trying to get the value out of it, not so much the pristine code.

41:35Sometimes that's the value part too, and you enjoy the process, but just for an exercise, because I want to automate Proxmox, why not do it via this Ralph Wiggum loop, which is to just do it until it's done. And you can kind of give it repetitions. You can say, okay, do, you know, one, you know, one version of it. Not a version, but I think like tries, I forget what the terminology is for it. Let me see if I can find it real quick. It's like one iteration, two iterations, and you can specify because you have, you know, only so many dollars to spend. I don't want to spend more than 20 bucks on this feature or 10 bucks on this feature.

42:11So either spend 20 bucks in the feature or, you know, 10 or 15 iterations until you get to some result. And then I'll come back and examine it and to run it again. All you do is just run it again. It's like item potent in that way. It'll just go back and do it again. Like that's such a cool world to be in, man. Yeah. Building a little home lab gardens with that kind of loop. So cool. Yeah. Yeah. No, that is for sure. Because, yeah, a lot of times, you know, agents will stop. And I know exactly what you mean now because, you know, you can tell it to do something to completion, but it's either going to stop or check in or do something.

42:47You know, and I know the prompts that I give. I have a lot of my prompts saved because they're, you know, annoying to keep explaining to AI, you know, like, you know, fix all unit tests and run all linting and do all this. You know, do these things. Don't do these things. Go, you know. And to be able to not be a human in the loop anymore until the very end is pretty cool to think about. It's like, no, you loop, you figure it out. You do so many iterations of this piece of software or you do one really good iteration. And let me see the final result. At that point, you're just like a director. You know what I mean?

43:23You're just like, you know, a director. That's right. Yeah. Give me some. That's right. A parent. Go clean your room. Don't come back until it's all the way clean. And you know what? I'm going to check under the bed. I'm going to check in the closet. So, you know, make sure you don't put stuff under the bed and in the closet. And when I come back in, you know, an hour, it better be done. Right. I'm not a parent, but, you know, I remember those days. You know, thinking I can outsmart my parents by GM and everything. You got your pups, right? That's right. Oh, yeah, yeah. They're pretty clean, though.

43:57You know, and they don't clean up after themselves. I mean, not yet. Well, generally, unless, you know, something. Yeah, I won't go there, but but yeah. What is the centerpiece of your home lab right now? Like what are the centerpieces? I imagine Proxmox and TrueNAS is still there in the center. Unify Hardware is obviously probably part of the center. What's what's around that center? What you're building on? It is. So, so this kind of goes into, you know, another one of my predictions for this year too, is if we could ever build anything is, is one big box. I think people are going to return to this one big box idea where, where they're, yeah.

44:37Only because things are hard to get ahold of. And, you know, while you might be able to get hold of lots of little older machines, you know, the one liter machines, you know, to do, you know, clustering, I feel like now that things are so scarce, people might be going back to one big box. That's your storage, that's your AI, that's your compute, that's your, you know, virtualization, that's your NAS, that's your everything. And it's kind of the way I've been going, too. So I do have my TrueNAS box. It's one big box, you know, has a video card, has RAM, has 10 hard drives, you know, GPU, all that stuff.

45:14And so that's, you know, not only my NAS running ZFS, but it's also where I'm running my applications now, too. So I've moved a lot of my applications onto my NAS. Yeah. I watched that video of yours where you were doing stuff with containers and you had to do that sort of sidecar load, which I did follow. But I didn't get the same results you did. Is that the way you're doing it with that whole, you have to create the YAML file and then it knows about it in the app container world? Yep. Well, if you're talking about TrueNAS, yeah, yeah. So you don't have to use YAML and do it that way. I do because I want YAML because I'm a developer, but also it's a lot.

45:49I'd much rather edit YAML than fill out a form any day, even if you could give me, you know. Same. Yeah. And so, and also because then you get CLIs and you get help from AI, you get all the stuff you get with YAML. And I can do it in VS Code. So there's that too. So yes, so I'm now running my applications on top of my NAS. So I've always gone back and forth. Like, you know, do I want my NAS to just be a NAS and just be storage? Or do I want my NAS to be an application server too, and then run those applications on top of my NAS? And so I've done both and I'm still kind of doing both. But for the most part, what I'm calling now my home production is on my NAS.

46:29And so my home production, you know, I've gotten a little bit wiser over the years and a little bit crazier. But, you know, I have a home production now and my home production is where the services are that need to be up. They need to work or I'm going to hear about it. You know, that's Plex, that's my NAS, that's, you know, whatever else I have running, which is a lot of stuff. And when I say I'll hear about it, I don't just mean my wife because she will say something if Plex is down because we record a lot of stuff. And if it doesn't record Survivor on whatever night, I hear about it. So that needs to be up.

47:04But also, you know, alerts and stuff I have set up and running, too. So, you know, my home production stuff even. That is right. Yeah. Grafana is on there. Oh, I've been getting so deep on there. Oh, yeah. I've been getting so deep in Grafana and Prometheus now. And it goes back to what you were saying. Like, you know, you know, I do a ton of, well, I've done a ton of Grafana and Prometheus in the past, a lot of observability stuff, you know, in the enterprise world or corporate world. But at home, I was always kind of like, man, that's, that's a lot of work. That's a lot of work to get that going.

47:34You know, now that I have help from, you know, LLMs, it's, it's work I want to do because while I could muddle through it and spend a week getting, you know, scraping, working on one machine, you know, the trade-off just wasn't there. And so now that I can scrape metrics on one machine in about 10 minutes, the trade-off is there. And so it's worth it to me. And so, yeah, tons of tons of I'm monitoring everything now. I have metrics on everything now. You name it. You name it. And I'm going to show off some of this pretty soon in my home lab tour. I do every year. It's coming soon, both the hardware and software, everything I host and run.

48:15Yeah. So that's coming soon. but I need to make it really good for YouTube. People have certain expectations and for some reason it's always got to be a little bit better or people will be like, oh, that's what you did last year. So yeah, so I'm running my applications on my NAS now and there is a reason for that, not just because I want one box, but ZFS is a really, really good file system. And when you layer on stuff like caching and then metadata, special VDEVs and separating out your metadata and putting that on ultra fast storage and then putting, you know, your app data on fast storage too.

48:54You get this really good bulk storage that can perform like NVMe storage. And so that's been my idea, this kind of crazy idea they have. I'm putting 10, 10, 14 terabyte hard drives in an array, you know, I'm doing Stripe VDEVs, kind of boring. And then on top of that, I'm augmenting the things that need to run fast, like metadata lookups and app data, and putting that on super fast storage. But anything in that ZFS pool can also use it. So it's kind of a tiered approach to storage. You have Oracle, which is RAM. RAM is going to be the fastest. Then I have this special VDEV where I can put files on there too that are below a certain file size.

49:40And then I have bulk storage. So my idea is run hybrid, hybrid ZFS, you know, all of my video editing goes on there, but also all of my databases still go on that pool too. And I still get, you know, NVMe like performance for most of the things that I'm running. So it's pretty cool. And so it's been a challenge for me to get that working. So that's why I'm doing it. And mainly because, you know, I always look at it like this. You know, if Plex is running on one machine, which it used to, and my media collection is on another machine, now I have two chances for it to be down. If I reboot my NAS or I reboot my application server, right?

50:20And so now I've doubled the chances of that service being down in my home. And so I co-located everything onto one box. So now it's like, well, you know, if the NAS is down, that means the apps are down too. But my NAS should never be down. And if it's down, we have big problems. So, yeah, that's kind of what I've done. I still have a Kubernetes cluster at home. You know, I still have three Proxmox in a cluster running Kubernetes. And I still have that on mini machines. That's kind of my home lab test kind of lab where I test stuff before I actually run it in production, which I have a co-location too, where I'm self-hosting Proxmox and another Kubernetes cluster there that's running technotim.com, plug for my website.

51:03Like that's all self-hosted in a co-location on hardware I own running on Proxmox in a cluster running on Kubernetes that I maintain myself. So, you know, I, my home kind of cluster is kind of a testbed for that too. So yeah, I actually run and manage three Kubernetes clusters. I have a lot going on, but it's fun. That's a lot of stuff to run. I was going to ask you about your clusters is because I went the route you did a while back when you built that cluster from the Nux. I'd only bought one because I could only afford one, but you bought three, and I thought that was cool. And I think that's where you ran your Kubernetes, or you ran Proxmox on high availability there, one of the two.

51:40That's right, yeah. So I don't have Proxmox in high availability. I have them in a cluster. I'm like, I get it why people run Proxmox. Yeah, I don't have that need to do that. Yeah, yeah, because I don't need HAVMs. I push that further down to the right, to the left. I don't know which way it is. I push it further down to the services and then run HA services, right? Like I don't need an HA Kubernetes node, right? You know, I just build more nodes. It's the whole cattle, you know, approach where I'm like, you know, it's great for, you know, if you have a single VM, something super important, that's old legacy app that you need to run and it needs to auto migrate somewhere.

52:18But with Kubernetes, you know, as you know, you don't worry about that. You just worry about the services. you run three replicas and if one no goes down, one no goes down. So that's the approach I take. I don't have Proxmox in HA. It's just clustered so I have one UI and I can migrate stuff easily. You got your TrueNAS, which is running your home production services. That's right, yep. You got your Colo, which is in a data center. That's right, and that's all of my public-facing stuff, yeah. Yeah. Which makes sense because you want bandwidth there and, you know, maybe no firewall poking, stuff like that.

52:55Although you could probably use tail scale or something else to do that. Yeah. Yeah. No, I used to I used to host it out of here. No problem. You know, update DNS dynamically. It was it was all fine. And I could today. I just wanted to kind of expand. And I had an opportunity from a local person here in Minneapolis to to join their colo. And I thought, hey, why not? Why not? You know, let me do some super fun, you know, site to site networking stuff and backups back and forth. So pretty cool stuff. What do you run on your Kubernetes clusters then? These applications? Like what do you run in there?

53:29So Kubernetes clusters. So I have, yeah, their applications, anything from Discord bots, websites. So I host, you know, my own documentation site that's, you know, multiple replicas. I have my own link site. I have some APIs that I run, two or three APIs, because I have this mobile app that I use, that I built many years ago, that's still running. And this mobile app then has, you know, APIs, which then needs, you know, databases. Databases are in there. Other people's websites, like, you know, my brother has a website, I built a forum, you know, my other brother has a website, I built a forum.

54:08So those are hosted in there. A whole bunch of I'd have to look But just random stuff But it's basically web dev stuff A lot of web dev stuff And Proxmox was mainly just skunk work stuff Like lab stuff then? Well those are So Proxmox is actually the host My Kubernetes nodes are VMs In Proxmox, right? So I'm not running Kubernetes bare metal I'm running Kubernetes as virtual machines So I have nodes that are Kubernetes nodes running on Proxmox. And then Proxmox is also running some LXCs like DNS, Postgres. So I have Postgres in a cluster running on an LXC because, you know, kind of mixing that into Kubernetes while it does work, depending on your IO, can go bad really quick.

54:59And I don't have a ton of IO. And then, yeah, I'd have to look at the list. but I use LXCs too. I was always kind of against them. I know they're doing containers now too, which aren't really great. It's kind of like a hack of how they're doing containers now. But I am using LXCs for small things. And when I say small, I mean like, I don't need a full OS for them. Right. Most of the time I don't. It's interesting the way you're using TrueNAS because I was always in the camp of let my NAS box just be a NAS box, but I'm kind of bummed because it's a Xeon CPU. It's a ton of RAM. And so I look at it, I'm like, well, you're not doing really much.

55:44I mean, like a file server is not taxing. As an enterprise, maybe with like thousands of users, that's the box you want. And so I've never really been happy about that scenario. But then I'm like, you know what? One problem, one issue, it's a NAS. I don't want to conflate what's on there because if I start putting applications on there, different things on there, well, then the uptime may go down or I may have an issue that is not NAS related. So then I've mainly been like thinking like NFS mounts. So why, what made you want to put applications there versus just NFS mounts? Yeah. So NFS mounts are great.

56:22You can get into some trouble with NFS mounts like SQLite. Yeah. Terrible. But not like other apps. They just need to have a storage, you know, like a database application. You want to be closer to the actual storage for sure. Yeah, yeah. And it's not even just the latency piece. It's the locks and everything like that. Like NFS just doesn't handle it so well. Like the latency I can kind of get over, but like SQLite in general, you'll have these locks that are like locked that you can't unlock and you'll get corruption and stuff like that. But NFS mounts were great. Yeah, I mean, you got to figure out permissions and stuff like that.

56:57you know and i went down that route you know too for a while but then it's like i have to back up i have to like you know take care of applications over here you know set up all those mounts and do all that stuff and then also still you know care and feeding for the nfs mounts and snapshots and keep that connection up and again you're back to you know you've just you know doubled your chances of downtime, you know, two, you know, it takes two, right. It takes two to make one. So, you know, and again, like I've gone back and forth with this, like so many times, you know, it's, it's, it's like this whole generalized versus specialized, you know, that you see all over it and, you know, enterprise in general, I've generalized and specialized my servers so many times that like, I think I'm ahead of like corporate, corporate, corporate entities that do this with their, with their, uh, with their, um, employees.

57:49Um, and so I've specialized and generalize my NAS so many times with so many different things. But right now I'm landing on this. And I think a lot of it has to do with two things. Well, actually three. One, TrueNAS ditched Kubernetes and went back to plain old Docker, which was awesome because I never would have done it if they were still running the whole Kubernetes bit. But they went back to just standard containers, quote unquote, standard containers, you know, Docker images, Docker containers, I guess I should say. Two, that I'm able to do it with the YAML because I'm not going to fill out their forms.

58:21If they ever take that away, I'm bailing. I'm going to find something else to do because just not me. I don't want to fill out a form to be able to put in my environment variables. With the.env file, I copy and paste them and they're there. Why should I have to do that in a form? I get why they exist because people aren't developers, but that's not how I want to manage my containers. But a lot of that has to do with Kubernetes too. And then The other piece is this whole like hybrid. Well, it has a lot to do with what you just said. It's like, Hey, you have this beast of a machine just sitting there doing nothing.

59:00You know, I feel like it's that meme where that guy's like poking that thing with the stick and he's like, do something. You know, that's kind of how I feel. If my NAS is like, it's kind of boring to watch the metrics. I was like, yeah, dude, tell me about it. It's funny. You don't know much over there, beastie. Oh, it's funny you say that. Cause now you should see my metrics. Cause now I, I post all my app metrics on the screen. Like you can put little widgets, you know, my, my, my, uh, uh, traffic reverse proxy, you know, it's five or six megabytes per second, you know, which is, you know, not a lot, but you think like that's all day every day.

59:32Yeah. You know, and then like, you know, I look at my, I can see my MariaDB database, you know, going, it's doing, you know, three, 400 megs per second of queries and stuff like that. I'm like, yeah, this thing is like, it's humming right now. You know, and if I look at the CPU differences, you know, you know, it's only a couple percent, you know, more than it was before. And so I probably have 50, 60 containers running on there. And again, like the reason why I ended up doing this is because I figured out a way with ZFS to create this hybrid pool. I mean, people in production probably won't do this.

1:00:05I think it's cool, but, you know, to make my hard drives, normal hard drives, you know, be as performant and as responsive as NVMe. So not only the speed that you get, but as responsive. And that's, again, like I've layered in NVMe to handle all of the quick writes, quick files, quick access. and for anything else that's going into RAM, which is another huge case of like, yeah, I want to store my applications on something that has tons of RAM. And if it's storing, you know, bits and blocks in ARC, which is, you know, ZFS's RAM cache, if it's storing that stuff in RAM, yeah, do it. I want it to.

1:00:48So I'm like getting like the super performant, you know, apps that are mostly reading out of RAM. If they aren't, then they'll read out of NVMe storage and then worst case scenario they read from a slow disk which is worst case scenario because i have all those tiers can you tease what your what your ram your vdev is which is mvme and then your your disks what's i know you're probably a person some of your future videos bubble but oh no this is no i've done videos on this too so so my um so i have uh striped vdevs so let's backup so for bulk storage i have 10 14 terabyte hard drives okay and of those 10 14 terabyte hard drives i'm doing uh striped v devs or kind of mirrored pairs where i have two that are in a pair that are mirrored and so which means i have 50 percent loss of my data so you got seven terabytes essentially yes per pair is seven terabytes yep that's exactly right and so but i I do that for two reasons.

1:01:54One is because you can, in mirrored pairs, you can stripe them across. So you're basically kind of getting like a RAID 10. I don't want to say a RAID 10. Think of it like a RAID 10 where, you know, you have these pairs, but the data is striped across. So you get the performance on reads and you get performance on write, which is good for me when I edit videos, because, you know, I have a bunch of sequential reads and writes, and I don't know where they're going to be. But on top of that, you can also expand by pairs. And so traditional ZFS is super complicated on expanding. And I know that they've been adding features to be able to expand VDEVs and do all this stuff.

1:02:33But, you know, when I started MyArray years ago, I realized that, you know, buying two drives at a time is a lot cheaper than replacing all four drives with four different size drives just so I can get a bigger pool. And so in the traditional sense with ZFS, it's kind of what you have to do is you got to plan up front and buy up front, but doing pairs, let me build incrementally and buy incrementally. So anyways, that's my, that's my slow disk array. Then I've done this thing called special V devs, which is basically you say, Hey, all of the metadata about that data, instead of storing it on the pool itself, move it off onto super fast NVMe drives.

1:03:17So if you end up in a folder, like I have some folders that have thousands of files, you know, that can take a really long time for it to parse that metadata and retrieve that metadata because it lives on slow spinning disks. Well, if you have a special VDEV, you move it off of there so it can look it up on NVMe. Then on top of that, another thing you can do with special V devs is you can say, Oh, by the way, don't just store my metadata there. If you find any files that are, I don't know, below 64 K, just put them on there too. That's pretty cool. So if you think about it, like now I'm now any small file, which usually takes a little while to find on spinning disks is now stored on there too.

1:04:02And so it's like, I've just given my whole entire array a huge boost because now most of the stuff is happening on NVMe. There's one huge caveat to that whole thing. If you lose that special VDEV, you lose all your data. But I'm listening. Yeah. Yeah. So you lose all your data. You're definitely a cowboy in this. I mean, but you know what you're doing. Yeah. I've done the same thing, mirrored VDEVs. So I have four NVMe drives, you know, and so two would have to die for me to lose all my data. Right. But then on top of that, you should have a backup. Right. And so, you know, I am definitely going cowboy because I want the performance.

1:04:39But I think that's, I'm being a careful cowboy. I don't know if that's possible. You got spurs on, man. And you got your gun on there. You got your slinger. That's right. But my slingers are on safety. How about that? You know, where, you know, I'm still, you know, taking precaution. I'm still building in redundant, you know, redundancies. Yeah, if you're a rocket one NVMe, I was going to call foul Tim. But you got four, of course. That's right. True Tim fashion. So not only four, but you got them in pairs. So you can lose two of the NVMe. Now, did you also go with different brands, buy them from different places, things like that too?

1:05:16or did you just buy a batch? So on top of that, there's these old Intel Optane drives. So Intel used to make these Optane drives. They're ridiculously fast, and they have like the lowest latency ever. Intel stopped making them. Yeah. But these things will like outlive Earth. You know, like these things, like the read and write performance you get on them and the longevity and how many times it can read and write is like ridiculously long that I ended up buying four of them because of how fast and because of how responsive and because of how long they're supposed to live. So no, I bought, I bought, well, I bought two from, I think Newegg and then two more from Amazon, like later on, they're hard to find, but they're still the best drive for that, that specific use case.

1:06:07And so when those die, what I'm going to do is just replace them with Samsung, you know, consumer grade. Um, I hope they never die only because they're, they're the fastest thing out there. They, they blow away any NVMe that's, that's on the market. How did you get to get four NVMe's in this world? Cause that's, that's a, that's different. It is different. So, so I used an adapter, you know, uh, PCI express card. So, you know, traditional NVMe's, this is going to get a little complicated too, but, uh, NVMe's want to use four lanes of PCI Express where they can talk directly, you know, to the CPU or talk directly to something.

1:06:48I can't remember. I'm not a true infrastructure person. I just play one on YouTube. So, but anyways. The mask is off. That is right. That is right. Hey, I coined the term infrastructure as a hobby. That's kind of my thing. IH or something, that's what I call Homelab. because my actual career is, I'm a software developer, infrastructure is my hobby, but I love it. But anyways, NVMe can use four PCI Express lanes. And in your motherboard, typically you'll have one big slot that's 16X PCI Express lanes. You have to buy a card that can actually split that out into four individual lanes, and then you can address all four individually.

1:07:32There's also this thing called bifurcation. And so you need to make sure that your motherboard can do bifurcation. And what bifurcation is, is exactly what I just talked about. It's able to split that 16X card up into four individual, you know, lanes, four 4X lanes. And your motherboard has to support it and the card has to support it. Server motherboards, generally speaking, like Supermicro ones, a lot of them do. desktop consumer boards generally don't yeah and so if you're going to do it on a board that doesn't support bifurcation then you can buy a card which is really expensive that can do it for you on the card and those cards are i don't know four or five hundred bucks maybe even more but if you get lucky and you have a server grade motherboard that does it that's where you want to do it so anyways that's how i get four mvmes and one pci express 16x slot and they're all getting forex bandwidth for each so and then you still have lanes for your gpu that's right yeah so i have a lane for my gpu gpu is in the primary probably right yep and maybe one of your secondaries which are bifurcating you probably have a workstation or a server grade motherboard i'm assuming because you got that ability yeah workstations are great workstation motherboards are a great hack to not have to go server grade and not go consumer grade you kind of get that middle ground you're still in the 600 800 bucks maybe a thousand dollars for the motherboard but But, you know, that's because it's just the world we're in right now.

1:09:05But you do get the capability and you usually get ECC RAM availability as well versus non ECC RAM, which you don't really need need in an ass world. But you should have if you want peace of mind, I guess. Yeah. Non-corruption. Yeah. That's like, you know, a lot of people will go back and forth whether you need it or not. People will say, well, if you care about your data, you do because you don't, you know, if the memory gets crushed. Well, that's where it writes first. It's the source of truth. Yeah. So in ZFS, it's less important. I'll say that because of how it checks and how it does its CRC checks on the memory and can verify the data.

1:09:46Some people still swear by it. Like they'll say, like, if you don't use ECC, you might as well not even do stuff. You know, people go to the extreme. I'm not there, but I'm glad I have it. How about that? That's an interesting world. So when you think about a problem like this, so let's zoom out to, you know, not scare the home labbers away, those curious folks. Yeah. When you think about this problem, do you get out your big old whiteboard? When you're specking this world out, how do you think about it? Because you're a YouTuber, too. So you think about it probably in the story arc. You also think about it as a technologist.

1:10:22How do you map this and plan this and test this world? How do you do that? You know, first and foremost, like I've been doing this for a long time, even before YouTube, like I've had a server in my basement since I don't even know, I'll probably date myself probably like 2004 or five, you know, going back to like this old piece of junk that when I was in tech support, I asked my manager if I could take it home so I could learn about Linux and learn about Active Directory. and he looked at me like I was crazy because the thing was, you know, 10 years old already. And he, you know, told me, yeah, you can take it, just take the hard drives out because, you know, I had data on it.

1:11:00And so I've been doing this for a long time. I've had a server in my basement for a long time. So I, I generally think about, you know, what do I want to do? What capabilities do I want to have? You know, generally speaking, you know, I want to, I want to provide file services. So I want a NAS, you know, to be able to do shares and store my files on. I don't want to store my files. Obviously on, on my system, I want to store them on my NAS because then that gets backed up. I want to have some compute, you know, and it takes very little compute to do most of the things you want to do. A lot of people will say, get server grade, you know, but I could spin up this I3 right over here.

1:11:37That's, I don't know, uh, from five years ago. And this will like destroy any, you know, self-hosted container I throw at it and like laugh at it. Like, so you don't need much compute at all. Ram is great, but again, like you don't need tons of Ram. And so I just try to think of, you know, what services do I want to provide? For me, it's storage. For me, we record a lot of TV and, you know, stream stuff at home. So I think about Plex. Plex then brings in a video card if I want to transcode. Most of the stuff I can direct stream here, but you know, when we travel or whatever, iPads, you know, phones, they want to transcode.

1:12:13So having a video card in there that can do that transcoding for me on the fly is good. You know, that's a decision point too. You could, you could get by an Intel stuff, but then I started thinking, well, I want to run models at home too. So I kind of want to, you know, have shared infrastructure. So I want my video card to be shared too. So that's what I do now. Share my video card with Plex, with Olama, with the stuff we just talked about, you know, with paperless AI and stuff like that. Technically that's going through Olama. But even some of my, you know, I do some transcoding for some of my, you know, video cameras too.

1:12:49So yeah, I try to find a video card that'll work in all those scenarios. And so, you know, I never want people to think like, oh, I need to go and buy this big thing before I can start home mapping. It's never like that. Like if you have an old PC in your basement, use that first figure out what you want to do with it you know use it as is if it had windows on it from 10 years ago wipe it put linux on it and if you're scared of linux that's fine you can put windows on it but i would say just try linux uh because you're going to find things are a lot more compatible yeah for sure and just just try it i mean you might find it sucks to be a sysad been at home you know uh i enjoy it i enjoy it when things go wrong my wife says something doesn't work, that's when I'm like, all right, you know, it's developing something broke.

1:13:42I have a job. That's right. This is, this is why I'm backwards. Oh, that's right. That's right. Yeah. Um, but you know, I, I enjoy doing it. You know, a lot of people joke on a lot of my comments, you know, on, on, on my YouTube video are bro has a full-time job at home and it kind of is right, You know, to an extent, but, but if you build things right, you got things working and you, you know how they work and you document it in case things go wrong and a little help from AI, like a lot of things are hands off. So you're focusing on the next thing you want to do. So, you know, again, like I don't have like, you know, a recipe of how I do things.

1:14:26I just try to think of, you know, what are my base services I want to have? It's always going to be storage streaming and some compute and some kind of trans code capability. And from there, I just try out a whole bunch of containers. You know, I treat containers now. Just think of like apps on your phone. You know, I mean, I can't spin them that quick. I can spin them up, you know, probably about five, 10 minutes, uh, all said and done working with a proper certificate. But I, they're like apps on my phone. If I want to try an app, I try the app. Does it do what I want? Yes, it does what I want.

1:15:00No, it doesn't. There's this other app that I should try. And so I'll go try that. And, you know, that's kind of the world I live in now is, you know, my self-hosted services are basically apps at home, uh, for my home to use. I want to say at home I mean pretty much me but you know there are some that my wife uses yeah yeah and you know and it's as long as I have a platform to do all of that it it really doesn't you know matter what parts I used um because you can get by with so very little nowadays they really can I agree I don't think you have to go so big to I mean like this is where it leads Okay.

1:15:38It's like, I don't know if you're a golfer, Tim, but that's what golf folks say as well. It's like, Hey, you get invited to play golf. You're like, nah, I don't want to go. And then somewhere during that first round, you're like, Oh my gosh, this is the best game ever. I'm going to go buy all the clubs I could possibly buy. And golf tech, like any tech is just limitless really what they, what they can like fine tune and dial in. And so if you've, if you've ever become a golfer, you start with like this small itch. And the next thing you know, you've spent$10 ,000 your first year in some way, shape or I'm being facetious.

1:16:11But, you know, golf rounds aren't cheap. Golf trips with friends aren't cheap. Golf clubs are not cheap. And then you have to have special clothes or you want to have special clothes because, hey, why not dress the part? I kind of feel like that's the same thing with the home labs. Like you can begin with like I did. I can remember the day when I got my Raspberry Pi. I can remember the day when I spun up the TrueNaz or actually wasn't TrueNaz. It was just like whatever 45 drives sent me. Cause like way, way, way back in the day, I want to say probably six or seven years ago, I mean, eight years ago, they sent me an AV15 to try out.

1:16:44And they said, hey, no, you can just keep it. It's just yours just to have them play with. And this is when they were first launching that line to home labbers or what would become home labbers. And I was like, really? I'm like, yeah, we don't want to back it. It costs way too much to ship it back to us. And at that time, I guess hardware was just cheap enough. They were like, yeah, we don't even want to back. Just use it and enjoy it and tell people about, you know, your experience about it. I'm like, okay, cool. And so that today is my TrueNAS box, you know, and it's got the Xeon Silver, I think 4012, I believe, the CPU in there, if I recall correctly, or 4212 maybe.

1:17:17But I began, that was gifted to me. I didn't even know what I was doing with it at first. But then I began on a Raspberry Pi and started to experiment. And so that's where I began with everything ran on my Raspberry Pi. Now I wouldn't because my, my, not my needs, like what I actually need for computers has grown, but my desire to play with bigger things has grown. You know, the, the playground, I can't just have the carousel on it. I got to have the slides. I've got to have the swing. I got to have the rope climb. I've got to have the, you know, all the things, you know, so my playground for home lab just grown a little bit.

1:17:49But I'm curious, though, when we look at maybe this potential dichotomy between TrueNAS and your desire to put all things there, which maybe somebody out there is having similar feelings. And then this world of Proxmox. I feel like I'm with you. I kind of want my TrueNAS box to do everything. But TrueNAS, the software, is not quite there yet. Where do you see, do you have this vision or purview into that world where you can see TrueNAS being this all one big box software? Because it traditionally has been great, you know, great for what it is. You know, ZFS, storage pools, a little bit of applications you need it, but not a tremendous amount.

1:18:35I kind of want my Proxmox and my TrueNAS in one box. Is that how you feel? Yeah, I do. And you kind of can. So you could virtualize TrueNAS. So let's take your thing for instance. I've done it for years, and it's totally fine. Totally fine. It's totally fine. It's totally fine, dude. It's totally fine. But don't you have to map them to weird drive IDs and stuff? Isn't it weird in the uptime, I suppose, if you had some major issues with your drives? No, so it's easier than you think. So let's take, for instance, you have that TrueNAS box. It's running TrueNAS. Just pretend you want to run Proxmox on it now, but then you create a TrueNance virtual machine and you give that TrueNance virtual machine, you pass through the hardware of that HBA controller.

1:19:24You give it the whole piece of hardware. You say, nope, this hardware controller, hard drive controller, HBA is now assigned to this virtual machine. And what that does is now the virtual machine has direct access to all of those disks. There's no IDs to pass through because it thinks it's the true owner of the disks. And then you do that, then your life is good. And then whatever, that's one way to do it. No, that is one way to do it. But I, I agree. So I, let me, let me put it this way. Like, and this is not a dig to either product. They're better at some things. And so, you know, TrueNAS is leading as a NAS.

1:20:01They're, they're, they're, they're leading with, I'm a NAS, but I could also do apps. And I can also do virtualization. Not that great. And that's my own opinion. But it can do virtualization. It's just not that great at all because it's not a hypervisor first. It's leading with NAS. And when you think about Proxmox, Proxmox is like, hey, you know, I'm a hypervisor first. Like, that's what we do. We do virtualization. You could install apps like LXCs, although I'll get into that, but you could install a whole bunch of apps and run them on the machine itself. But, you know, and that does work. But like, if you think about it as a NAS, not great NAS experience.

1:20:41Sure, you can install a Samba server, assign it some, you know, pool, and then do all of the Samba config and CLI and all that stuff like that, figure out permissions. You can do all that. Like, they're both capable of doing each other's job. I'm kind of like, I want something in the middle. And there is HexOS, which is kind of coming, but that's kind of like, you know, some joint venture between some people and Linus and, you know, TrueNAS or IX Systems creating this more, I guess, consumer-friendly version of TrueNAS. there's that coming. But, you know, I played with the beta looks pretty cool.

1:21:25It's it's kind of going to be, you know, a facade on top of TrueNAS because, you know, they're going to use TrueNAS APIs and it's really TrueNAS under the hood. But that's something different. I want the best of both worlds. I wish they would. I mean, they're not going to do this. But if I if I could design, I guess the perfect NAS at home for me, it would be, you know, it would be like a true NAS-like experience for the NAS piece and maybe even for the application piece. But give me, you know, give me the virtualization capabilities that Proxmox has, you know, and the networking capabilities that Proxmox have.

1:22:01Or, you know, if I couldn't do that, I would love for Proxmox just to run Docker containers. They are so, I don't know what's going on there. They're so against like running Docker on the host. And I know you could shim it in and do it yourself, but they've even got to the point where now they're like converting, you know, OCI containers, which, you know, a Docker containers, quote unquote, to LXCs to then run as an LXC because they don't want to run true OCI containers like Docker. Like I don't get it. Is it licensing maybe? maybe i i mean i would have it's licensing i i don't know docker ce does it does it cost money to i mean i don't know i'm i know that there's there's been a lot of change in the docker world in the last you know six years i mean they were almost a dead company and then yeah uh then they were revived it was docker 2.0 uh i talked to their ceo on a podcast here on the same podcast you're on a few years back.

1:23:06It was a fun conversation about Docker 2.0 where they went from zero revenue to revenue. And I mean, at some point you have to protect your moat. Really, you do as a company. And I imagine it's probably licensing. It's probably something there. But, you know, as a user like you are, I want that world to marry. So figure it out. Yeah, yeah. Like, and Docker's done, like now that they're, you know, they have model runner, they're doing all this scout stuff and scanning containers and like really building up their Docker desktop. And I'm just focusing on the Docker CE part. And maybe there is licensing around, you know, maybe you can't ship this with your own product, but let's take that all the way out and just go down to like container D, like just get, you know, OCI images, Podman.

1:23:49I mean, something like at the end of the day, I don't want to run - True, Podman is available, more license friendly. Yeah, and so at the end of the day, like I wish I could run OCI containers as first-class citizens on Proxmox. And I, again, like, I don't know the reasoning behind it. I honestly feel like it has something to do with more of their strategy, you know, and how they're, you know, trying to be highly available and LXC is highly available, although that I don't think is there yet. And VM's highly available. I just don't think they want containers spinning up on the host itself. I get it.

1:24:24You can get around that.

1:24:35Friends, you know this, you're smart. Most AI tools out there are just fancy autocompletes with a chat interface. They help you start the work, but they never do the fun thing that you need to do, which is finish the work. That's what you're trying to do. The follow-ups, the post-meeting admin, the I'll get to that later tasks that pile up into your Notion workspace looks like a crime scene. I know mine did. I've been using Notion Agent, and it's changed how I think about delegation. Not delegation to another team member, but delegation to something that already knows how I work, my workflows, my preferences, how I organize things.

1:25:08And here's what got me. As you may know, we produce a podcast. It takes prep. It's a lot of details. There's emails. There's calendars. There's notes here and there. And it's kind of hard to get all that together. Well, now my Notion agent helps me do all that. It organizes it for me. It's got a template that's based on my preferences. And it's easy. Notion brings all your notes, all your docs, all your projects into one connected space that just works. It's seamless, it's flexible, it's powerful, and it's kind of fun to use. With AI built right in, you spend less time switching between tools and more time creating that great work you do.

1:25:40The art, the fun stuff. And now with Notion Agent, your AI doesn't just help you with your work, it finishes it for you based on your preferences. And since everything you're doing is inside Notion, you're always in control. Everything Agent does is editable. It's transparent and you can always undo changes. You can trust it with your most precious work. And as you know, Notion is used by us. I use it every day. It's used by over 50 % of Fortune 500 companies. And some of the most fastest growing companies out there like OpenAI, Ramp, and Vercel. They all use Notion Agent to help their team send less emails, cancel more meetings, and stay ahead.

1:26:16Doing the fun work. So try Notion now with Notion Agent at Notion.com slash changelog. That's all overcase letters, notion.com slash changelog to try your new AI teammate, Notion agent today. And we use our link. As you know, you're supporting your favorite show, The Changelog. Once again, notion.com slash changelog.

1:26:40Do you mess with fly.io by any chance? That world in like prod, fly.io? No, I haven't. I mean, if you love containers, then you'll love fly. I mean, Fly is what we host on changelog.com. They're a partner of ours. We love them, obviously. This is not technically paid. I'm not paid to love them. We just love them anyways. Fly is like that. You're running containers, right? You're running a container in production. It's Firecracker VMs. I'm not familiar with everything behind it, but Fly machines, essentially. They spin up very fast. They run down really fast. you know i want i would love to have a version of fly in my own lab and it sounds like that's what you're describing there which is i want an oci container i want you know as close to bare metal as i can i don't have to spin up a ubuntu vm to then throw docker on to then launch my docker container i would like to just have the entire system be container friendly it sounds like you're saying yeah yeah yeah i mean you know similar to cloud run and google or any of these things.

1:27:43You give it a manifest, you spin it up and it, you know, it's, it's running, you know, container only. And I mean, that's kind of what I'm getting with TrueNAS right now. I feed it some YAML. I got to create a dataset for it, but I got to tell it where to put the data it's going to use. Then I use a Docker compose file and I'm done. So that's kind of what I'm getting with TrueNAS now. I just wish, you know, Proxmox would just build that into their CLI or UI or whatever so that people just don't have to do this. Well, I'm going to run an LXC and I'm going to run it as root so then I can install Docker to then like run, you know, an OCI container inside of this LXC container or, you know, or pay the virtualization tax and run it inside of a VM.

1:28:24I just want, you know, as bare metal as possible. And I don't know, I'm sure Proxmox has a lot of reasons why they don't do it. Probably, you know, has something to do again with their, their, their strategy. It doesn't fit in there, but I don't, I just don't see how you could ignore OCI containers in general. Like, yeah. Yeah, I agree with that. I do agree with that. Well, the closest I've been able to come is the CLI I built, which in this case, it's got the cloud image, an Ubuntu, or even a Fedora cloud image on the server already, which it uses that as its base, like you would. And so rather than going that whole route of creating the template, including the template to create a new machine, the CLI does a version of that through automation.

1:29:10And, you know, you're able to, you know, through CloudInit, you're able to define the network, right? You've got all that there, the user, the network, the SHK, and then everything else is the Proxmox API, which I really wish Proxmox and TrueNAS did a better job of documenting their API. It's just not, it's not super, I mean, it's good documentation, but I just feel like they don't treat it like a first-class citizen. And me, the kind of developer I am, is I want to play with your API. I want to build my tools on top of your system, not be forced to go to your web UI, even web UI. Like even with TrueNAS, like you're the same.

1:29:48I don't want to fill out a form to spin up a new thing. Well, I would much rather automate through whatever layer it is, whether it's me or an agent, some sort of CLI. And then if the agent's using it, then it can just easily use the CLI I built. But I want to be able to automate those things on those kinds of systems. And the closest I've come is that exactly. It's like PMX, VM, new, and then specify all these things, send it a template, which is super easy. And those are YAML files. Those are YAML files, like two or three YAML files to define a couple of things. And you're off to the races. You can define a minimal Ubuntu brand new machine.

1:30:31Now, you could define it to be bigger, but I just have found that it's just more easy to layer on a post-install Bash script than try to script it all. Like it got into Ansible land. It was just really nasty. It got really error prone. So I was like, you know what? Forget all that. I just want to define a base VM. A base VM that is blessed with an IP address, with the RAM I want, with the CPU I want, with the disk I want, and protect it. you know it's got a dash p on it which means if i try to delete it accidentally or my agent tries to it's got to go through this whole entire dance and you know send across like documentation and social security numbers stuff like that to like delete a vm like you can't just accidentally delete a vm you know what i mean like it's you gotta do some things you know uh but it's pretty i mean like literally within so your 15 minute scenario that you defined earlier with a new container less than 60 seconds Tim yeah maybe even 30 maybe a minute to get the IP address back because it's got to like launch the VM get all the updates right from a boon tour whatever which takes time and then actually launch the actual machine itself and then get an SSH key that's the thing that takes time is the boot the update and then the finally you know QEMU giving you that IP address back, but like, that's all like instant.

1:31:51Yeah. That's, that is awesome. And so when I said 10, 15 minutes, that's me figuring out, okay, data mounts, environment variables, like that's me. That's like research I had to do anyways. If I, if I knew it, yeah, it's still take me five minutes, but that's not 60 seconds. I should, I should look at doing that with, with TrueNAS2. It's just automating. I'll show you what I'm doing behind the scenes. It's ready to be used if you don't mind warts. It's not open source ready yet because I think people have a different expectation of what that might be. I want to do one more rewrite because I now have a more clarity on what it is.

1:32:25But I think we could layer on what I'm already doing with the idea of already saying, okay, there's true NAS on the system here. And there's a separate layer that you can spin up that same VM. But then also say there's mounts elsewhere. There's NFS mounts elsewhere, whatever you want to do to do your world, which you're talking about there. So, yeah, I'm curious why you don't use Ansible, though, because I have my I have my post Ansible. Like I have bash, bash and agents, agents like bash. Yeah, I just mean like, no, I know I get it. But like I have my Ansible playbook for new VM, you know, and has 50, 60 tasks that are all there.

1:33:07Yeah. And and it does it intelligently. like, Hey, if I say, you know, stop the firewall service. Well, if it's not running, it's not going to stop it, you know? And Hey, if I tell it to install this one package, it knows that doesn't run on this type of machine. So it's not going to try, you know what I mean? Like, like I have my Ansible playbook where it's, I just click go. Like anytime I create a brand new machine, I have a standard playbook I run and it's going to apply updates, update, reboot, install Z shell, configure Z shell with Robbie Russell. Cause that's what I like, you know, it goes through the whole shebang of like, this is my standard VM.

1:33:43And I just don't even pay attention to it. You know, a couple of seconds, you know, it probably takes, you know, a minute or two throughout all the reboots and applying and installing packages. But after that, then it's like, it's ready, you know, ready for production. Yeah. So I think the, I'll, I'll defend that by saying I have never been, not for any reason, an Ansible guy. I just never really got into it. I I didn't understand the world. I know what it does and how people use it. I get all that client server. I get the recipes. I'm not foolish, but I never got into it to need infrastructure automation.

1:34:20And the limo franca of agents is Markdown files and bash. Bash is everywhere. You have to add on Ansible in your client or on your server somewhere. And it's baggage to native Linux, right? So it may work for you because that's been your history. But in that world, when you're trying to automate that kind of thing, I just tell the LLM, hey, I'm launching a new instance of DNS hole. And here's the specifications for it. It will write a script. That script can be item potent. And it will rerun it. And it's just as good, if not probably better and faster than Ansible. Oh, yeah, yeah, yeah. I'm not, I'm just thinking about like, if you're building, you know, if you're building a CLI, like at some point, like, you know, it has to, it has to be able to scale to different things.

1:35:13And so like, are you recompiling the CLI every time it needs to install one more thing or just writing a new script or how does that work? You know what I mean? Well, it has versions. It's written in Go, so it's got versions. and so if there's new features or a patch release i'll patch release it and you know throw a new version out there which you can then you know run pxm update and it will get the new version of it sure i just mean like the the source of truth for your list of things you know say like one day you want to install z shell like here's the beautiful thing tim those aren't in there yeah that's what i was going to say those things are in user land those templates are in user land you can define them and so that's why i want to rewrite it because i sort of like you know i haven't been doing this as a day job this is my little scratch my little itch and i really haven't i probably haven't changed the code in three months honestly like it does what i need it to do this adam wants to change it to release it and make it better but the world i wanted to build was this really interesting cli but most of those things live in user land and so have a separate repository that the world can define their own minimal ubuntu minimal fedora minimal you know debian You name it, whatever you want to do, and all the patterns are there.

1:36:24And all you've got to do is clone the repository, put it in a certain place, update your config. And when you run PXM, you know, new VM or whatever it is, and you do dash dash template or dash dash package and you say the name, well, it knows that because config says all of your templates are over here. So you won't have to recompile the Go binary, the Go CLI to do that. All that will live in user land. Originally, they did. Okay. I was a fool. OK, I put those things in there and I was going to make it to extract out. So I want to give people a good bootstrap. Well, then I learned, well, that's just probably not the best way.

1:36:57And so it's better to have a user land repository that people can commit to and update and it can be a pointer to people. And sure, it's one more step to clone that repo down and put it in a config. But I feel like the tradeoff is better long term. So that's where I'm at. I haven't gotten to the point where now I can put a lot of those minimal templates or even more expressive ones that use Ansible, which is totally possible. I just didn't want to do it. And I just wanted to get a base image and unleash my agent on that image and be like, make the world. And it goes and it makes the world, you know?

1:37:30Yeah, no, that sounds awesome. No, it sounds awesome. I'd love to check it out. Because, yeah, again, it was like, you know, I did an Ansible only because like that was way better than running, you know, CLI commands over and over and over. or even writing one big bash script. Now that's changed because LLMs love bash. So now things have changed. But yeah, I should definitely revisit it. Yeah, I'd love an agent. Yeah, yeah. At some point, I'd love to hook up my open web UI. So my internal chat that I use with Olama and build, use open agent or something is that new open agent that's out. Open code.

1:38:07Yeah, open code, sorry. Yeah, open code, run that agent and tell that agent to do stuff in Proxmox for me. All via chat. Like, I just don't even want to run a CLI. That's what I'm saying. Like, I don't even want to run the CLI. Sure, I could do that and expand the variables and figure out what it can and can't do. Well, the problem with it is that Proxmox has some challenges. Like, it would have to navigate. The reason why I went the CLI route was one, I thought I wanted it for me. Now, the whole world's shifted to be agent first. Yeah, I want to build an MCP. I want to build an MCP for Proxmox.

1:38:39that the MCP understands the Proxmox API. So, hey, MCP talk, you speak Proxmox API, LLM, you speak human, but you also speak to the MCP. You know what I mean? And so now it's like, I speak human to, you know, an LLM that speaks MCP to that, you know, LLM kind of API to Proxmox and just do the stuff. You'll get to a limitation at some point though. So the CLI is the blue layer in the middle there. For sure. So the CLI is kind of the important piece. and then you layer on the mcp server on top of your cli and so the mcp can speak native proxmox as necessary like if there's an api and using native api or the things that it doesn't do that you've taken from 15 steps to one command yeah and you've got tests against it well your cli in addition to the proxmox api with your mcp server and your agent in open code or in clod well that's well that's the beautiful world yeah man yeah that's yeah that sounds awesome that sounds awesome for sure because yeah the the proxmox binary whatever it is pxm i don't pxm that can do so much i mean that's how people are doing even their ansible scripts right now they're like you know shelling into it running a pcm command so yeah it's uh it sounds awesome for in MCP too, to be able to say that, like, but then that's a lot of words for me to type, you know, I'm getting lazy developer.

1:40:06Like I'd rather like you have a CLI and run a command. Like, do I want to sit here and describe, you know, what, what to build? No, just build a standard VM. Let me know when it's done. Call it this. Right. Well, the determinism there is the challenge and you can do that, but you, you know, the reason why MCP came into play was because LLMs are non-deterministic. You can say that, but every time you might get a different version of it. Or if you're using Sonnet versus Opus now, and like Opus 4.5 is the default, you know, per Anthropic and Cloud, they want you to use Opus 4.5. It's non-deterministic.

1:40:41It may know that and you'll get great results every time, but it's not the same path to roam. The MCP server is what helps you determine and create determinism or even skills. So you layer on skills, MCP and CLI with a decent API, which is why I would love ProxMonks to just bolster that, Make it more expressive. Give me more in there. Better document it. Because if you give us those tools, we'll use your thing more. And I got to imagine if I'm an enterprise buying a support license, that's how they make their money, right? Isn't that how Proxmox makes their money on Proxmox? Oh, yeah. Support licenses?

1:41:19Yeah, that's how most open source are making it. Now they're starting to do premium features. It seems like that's a huge trend now. It's like, oh, you can self-host the normal version, but then premium features. You know, it's a combination of I'm either going to be your SaaS, I'm going to give you extra features, and or I'm going to be your support. That seems how most open source projects are monetizing now. In large installations, I can see that. For me, I would buy a support license as a means to sustain and just to get rid of that thing that pops up every single time. Really, I would just be like, I would honestly give Prompt's Box 100 bucks a year just to get rid of that, you know, realistically.

1:42:04Yeah, there's a yeah, I agree. I asked them one time, hey, do you have a HomeLab license? You know, do you have a cheaper version of the Homelab license, you know, that I could use so I could get, you know, legit updates versus bleeding edge updates? Because their price per core or whatever it was, was, you know, kind of cost prohibitive for a Homelab person. It's like five, 600 bucks. Might be more now per year. And I'm like, yeah, that's a lot. Not a lot for enterprise, but a lot for Tim or prize, you know. So, and they were like, don't worry about it. You know, they were like, don't worry about it.

1:42:38just, you know, use our latest updates. Anyways, I say that because you can get rid of that nag screen really easy. You can. I don't know how to do it. Oh, man. So since we're talking so much about Proxmox, have you heard about Proxmox helper scripts? No. I did a video on it. It doesn't matter. You should check out Proxmox helper scripts. I pay attention to your channel, Tim, and I missed this somehow. That's all right. That's all right. No, it's going back to algorithm. them you know it's it's fighting for ears and eyeballs and you know and and google says like hey this does not deserve ears eyeballs and plus like dude there's no way you can keep up anyways that is a collection of tons of scripts single liners you can run to do anything you want to do on proxmox and all written in bash there's one that's like get rid of nag screen that you can run but there's also like the the default script you run is so good it'll like get rid of nag screens, set it to, you know, remove the enterprise repositories.

1:43:38You know, it'll do everything you want to do as a home labor, which is give me bleeding edge updates because that's all I can get. Turn off nag screens because I can't really afford a license and, you know, disable some other things because I'm never going to use it. So really cool, really cool repository. It was actually made by someone who passed away and he passed it on to the community. So really cool website and a really awesome group of people. What is it called again? One more time? Proxmox Helper Scripts. Okay. It might be proxmoxhelperscripts.com. I'm not sure but if you check it out, it's so awesome.

1:44:18I mean, it's worth looking at. Proxmox Helper Scripts. Looks like a github.io website. easily search that on the web we'll link up in the shows of course I know we're getting close to our time Tim I'm happy to keep talking to you but oh yeah go to view scripts I mean anything so they basically built almost like an app store for LXE containers which is pretty cool like hey do you want to install do you want to install home assistant as an LXE container on your Proxmox yes click one shell script you're done do you want to install Well, no, yeah. Do you want, do you want Olama with GPU enabled? Yes.

1:45:01Okay. Run the shell script. Yeah. Go to Proxmox VE helper scripts. It's community-scripts.github.io. And that's what they've done is they basically built almost like an app store for Proxmox because they're able to either create an LXE for it or create a VM for you. Like databases, like, hey, do you want to run Postgres? Yes. Do you click this button? run the shell script, you know, and you're done. This is cool, man. Yeah, it's really awesome. I did a video on it. I had no idea this even existed. Yeah, man. It deserves so much attention. And it's not building VMs. It's building LXCs for the most part, because that's what you want, right?

1:45:40You don't need a full fat VM to run, you know, MariaDB. You just need MariaDB running, you know, with a little bit of storage and a little bit of RAM. Yeah. Yeah. Like, like, like think about setting up, you know, Bitwarden, how hard that is, you know, or Othalia, you know, this does it for you. And, you know, very simple shell script. Well, all right, then. I'm glad I talked to you, Tim. Yeah, man. Like Home Assistant, like you want to run Home Assistant in LXC container? You just want to try it out. Click the script. Two seconds later, it's running. You don't like Home Assistant? Delete it, you know.

1:46:18Are you using this for those? You're probably not using this for those things that you're using it just for the simple things, though. It sounds like. I am using it for some stuff. I used it for PyHole instead of PyHole as an LXC container. Cause I'm like, yeah, why not? Like, do I want to run, you know, sudo apt install and do all this stuff? No. Do I want to run it as a VM? No. So yeah. So I've done it for, for certain things, for certain things in my lab. Yep. Right. Did I do it for Redis? I might not have done it for Redis cause I run Redis in a cluster. I do run Redis as an LXC to run it in a cluster, but I don't think I use their scripts, but anytime I, you know, anytime you want to test something out, I mean, this is almost faster than trying it with a Docker container only because, you know, they fill out all the defaults for you.

1:47:06So you go from, you know, you go from running the script to it actually running the challenge here. And the one thing, the big takeaway is here is you don't run this script in a shell prompt. it's kind of weird you have to run it from the proxmox terminal web that make sense you gotta log into your web ui yeah and open terminal things might have yeah things might have changed since then but if you run it in your shell you know you're not executing it i think is the right person so um and that might have changed there might they might have made changes but i know for sure if you run it in the terminal from the web, it works.

1:47:50So yeah, set up, set up Grafana, set up Prometheus. It's all right there. Wow. Yeah, it's pretty awesome. That is pretty cool. I didn't even know this existed. I've been using Proposbox forever and learn something new every single day. What's left? What's left on your, on your thought list? I know you made a list. I did make a list. Anything else on your list is like, man, dude, we can't end the show without talking about this. No, it was my, a lot of it was just, you know, I don't want to say complaints, but you know, just, just the state we're in with how expensive things are. Nothing's available, but software is getting better.

1:48:28So that's a huge upside. And, you know, people are making awesome things using a lot of tools and getting ideas outside of their heads. It's a lot of awesome cell phones and stuff in the open source world. And it's, I love it. I love it. I went from this, I feel like I went from this drought of software, I don't know, a couple years ago where I'm like, yeah, you know, I've seen all the Docker containers people run at home. So now I'm like, oh my gosh, there's this whole like new world of people building these new containers of things I can run that I've never even heard of before, you know, based on AI or maybe, maybe not.

1:49:04that like I feel like so refreshed because now I can try all these apps again. So yeah, these are like, for me, it's like server apps. Think of it like that. So I think this year's gonna be a wild year. I would definitely encourage you to unleash Claude on your UDM, whatever you might have. And just say like, help me just examine my VLAN scenario, examine my rules and my profiles. And it might be like, you know what, Tim, you're pretty good. Or it might be like, you know what, Tim, I can help you here. I was probably going to say like, why do you have these duplicate rules? Like I, there are probably rules in there that I either never deleted.

1:49:44Like I am not a firewall rule expert. I'll be the first to say is like, I try it until it works. You know what I mean? Like it's, it's, it's always guests for me. let me just say that I set up my VLANs based on your video and it was upset with me. Just saying. It's operator. Just saying. and i really didn't know much about him because i was like you know i've been told hey if you run a network you should have vlans because you want your kids to be on one thing which i totally agree you know you want your guests to be on another and i totally agree with that too and then everything else is on this trusted network which i totally agree with but then all this intermingling i'm like well just because just because it's an nvidia shield should i put it on my iot thing and then the answer is no the answer should be it should be entrusted it just needs access to too much, you know?

1:50:32And there's other things that like, it's the dumb things that you definitely don't want on your trusted network on the IOT VLAN. And so the three VLANs I'm for sure I want is trusted kids and IOT and guests. Yep. Yep. Yeah. That's, that's kind of where I've settled too. But then I have all of my networking equipment, you know, on a separate one. Like it's, you know, it's, it's on the default one. My trusted is its own VLAN. I do have one more and then cameras, all my cameras go on one VLAN. But yes, I agree. You got to think about like, it's not necessarily what the device is. It's kind of the role that it plays and how much do you trust it?

1:51:12And so some people might say, yeah, put my HomePod on IoT. I don't want my HomePod, you know, that's IoT. Well, if you give it access to your schedule and you're telling it to turn on your lights in your home, I think you kind of trust that thing, you know, enough to put it on your trusted network. And so that's the way I kind of think of it now is like, what does it need access to and how hard is it to go to, to, you know, cross the chasm, you know, like home assistant, like I do trust that thing, but I don't want to write a thousand firewall rules for it to go talk to everything in IOT. So I put it on the IOT because I don't want to do the opposite.

1:51:51So yeah, it's even my Xbox for a while. I was like, die hard like no you are iot but i'm just like you know i put my password into this thing and i play games it's it's gonna go on my trusted network because i don't want to cross the network just to do other things you know i think you just gotta worry about if that thing ever gets circumvented that's the main thing is like if it gets yeah yeah it's it's all about limiting your blast radius limiting your blast radius and how comfortable you are with that you know yeah i think this year though is the year of AI in the home lab. I know it has been already for me and I do lots of stuff across machines with Claude.

1:52:32I don't just do it on the single machine I'm on. I'm not only using it to build software or to build little itches and scratches and stuff like that. It really is like the moment, for example, when PXM gives me that IP address back, I take that info report it gives me, which I've designed to be agent friendly. In that case, I'm the CLI or I'm the API. I copy and paste it into the agent. I say, here you go. Here's your machine. And then it logs in because it's got my sh key and it's like, okay, sweet. It's a brand new, brand new, you know, base image of Ubuntu. Let me build your world here for you.

1:53:10Here's the bash group you want to write here. It stores it in, you know, in Git, in the repository and maybe a deploy file. or deploy directory. And we always make it idipotent, you know, so that way if we want to rerun or something like that, or if it needs a different one, maybe a post or a pre-install, who knows what, but that thing has just been so cool to just unleash like that. So I think this year will probably be your year too of AI in your home lab. And that's kind of fun. New worlds, new capabilities, Tim. Yeah, I definitely need to unleash some agents here and try some of that too. I've been, you know, I do run a ton of AI stuff and I've been, you know, doing a lot of LLM stuff, especially.

1:53:53But agents, yeah, it would be cool to turn some stuff loose in my Proxmox cluster and just say, go build some stuff rather than using an Ansible playbook. Just go for it. Rather than one year later, Tim, we should talk in three months. My prescription, if I'm your doctor, Dr. Stachowiak here, okay, is your prescription is go unleash Claude on your home lab network and do some cool stuff and come back in three months and tell me some tales. Okay. Because I guarantee you, I guarantee you come back a whole different, Tim. I bet. You know, it's just like, again, like a lot of people probably going through this, like, you know, I've had the tools that I've used and I've designed the tools and I know they work.

1:54:36and I'm comfortable with them and I write CICD pipelines, but maybe I should just kind of just give that to AI and tell it the outcome I want and not worry so much about how it gets there. Especially on the low-stakes stuff. You know, if you just got this little thing you want to do, what's the harm? You wouldn't have written it anyways. Who cares about the code? Why do you need to do a code review? All you need is a code viewer. Your VS code is now, that stands for code viewer now. I'm just kidding. I'm just kidding. it stands for continue continue continue right yeah exactly well i mean you can also automate a lot of that stuff too to be in yolo mode so a lot of people will say yolo mode for some things that's kind of ralph wiggum that loop there the rock the ralph loop is if you if this is your first time hearing about it tim you're going to hear a lot more about it soon because it's the it's the the beginning of what's going to come when it comes to a well-engineered like it doesn't remove you as an engineer you just have to be an engineer so be a good engineer with a good to-do list or a good spec and unleash ralph that ralph loop on it and you know if the code works and it passed tests and these different things around security well then who cares really in the end is it the best idiomatic go i mean i kind of do if i'm if i'm like maintaining this thing but if the agents maintaining it and I didn't have the software yesterday.

1:56:01I need it today and now it's here and it solves my problem. And I don't have the time to maintain it anyways. It's like, did the tree fall in the woods and you hear it anyway? Like it's that whole thing, you know, doesn't really matter. Well, kind of related to that. One thing I did learn too is, you know, after having AI write some code is having it write test too is super important. I've noticed. Yes. So I've noticed like anything I want to keep, have it write tests, because not only does it prove that it works, it's a good hint for it to understand how the code works. Just like humans, you know, anytime, like when I review someone's code, I'm like, where are the tests?

1:56:39Not because I'm asking where your tests are, because in your test, I can kind of figure out like what you were thinking when you wrote this and what you're trying to do. So I've noticed that too, is like anything you care about, you know, that you have AI writing, have it write some tests too. It costs you what? Some tokens? Not really any brain power, but tell them to do it too, because it's good. It just helps your agent understand in the future what it was doing before. YouTube.com. Do you say what your YouTube is out loud? and just techno tim i usually just yeah just people google techno tim uh it's techno tim.com now huge huge change no more techno tim.live it was that you know i i paid the squatter now i'm big time i have a dot com now i had to put tons of redirects in place i was like redirects everywhere actually i did them on the edge on cloudflare most of them but then i had to like find my old links then i have link shorteners then my email uh and then i had to Set up aliases and cut that over and a lot of DNS stuff.

1:57:43I actually did it in about a day. I had help from AI. I did about a day. So it wasn't that difficult. It was just a lot of things to remember. A lot of things to remember to do. And still, I think I still broke something. But at least I didn't lock myself out of my own email. So if you ever change domains, it's a lot. Do it as early as possible or never at all. Yes, my gosh. It's like renaming something. It's the worst, especially if, yeah, it's just the worst. Pick a good name from the beginning. Do what you can to never have to change your domain ever, ever, ever. Yeah, yeah, yeah. Never name your business after a street or a product unless it's like Main Street.

1:58:34Yes. I noticed that with so many local businesses. It's like, you know, we're, I don't know, we're whatever Boulevard, but they're not on whatever Boulevard anymore. It's like, oh, wait, that's making sense. Yeah. We actually have here in Dripping Springs, we have a Mercer dance hall and it used to be on Mercer. And it's not anymore because Mercer's real estate got more expensive. Now they're on like Route 12. Yeah. See, I was like, well, you're not Mercer dance hall anymore. It's like, where do you go to Mercer dance hall? Route 12. Everybody knows Mercer street here. You know, it's like, it's like, well, I'm here at Mercer.

1:59:09Where's the dance hall? It ain't there anymore. It's somewhere else. Yeah. Yeah. That and a product like, you know, yeah. Or like a price, you know, dollar store. They're not even dollar store anymore. It's like dollar 25 store, you know, but there's still dollar store. That one's kind of generic, but you know, if it was like, everything's a dollar or everything's$5 and it goes up to 10, then you're, you're in big trouble. Well, there was another story called something five. Oh, a five below. Five below. Yeah. You can go in there and buy things above five bucks. That is right. What is up with it?

1:59:42Yeah. Now, I get it. The color is blue and they make it like it's cold. But that's right. Like, you know, your double entendre is a single entendre now. You know, like, come on. You know, that is right. Yeah. Yeah. I get. Yeah. You better keep a five below in there, you know, temperature wise, because, you know, not everything's below five dollars. That's right. That's right. Well, everyone, TechnoTim.com. Check that out. Thank you, Tim, for... just exploring this fun world of home lab with me every year. but my prescription is go away. And instead of coming back a year later, come back in three months and tell me about your new world.

2:00:20I want to, I want to see Tim's new world in three months when you just unleash AI, even more so these agents on your home lab. I'm here for it, man. I love talking to you too. It's, it's, it's always a pleasure. You give me so many ideas and now I think I have a lot of ideas. I want to do something right after this next video, but yeah, I'm excited. I'm glad to be here. And it's always nice talking to you, man. Yeah, same, same, Tim. Same, Tim. Good seeing you. Glad you're well. Bye, y 'all. Bye, friends. Bye, friends. A new year. Fun time with Techno Tim. Always fun digging in and talking about the future of HomeLab with Tim every single year.

2:01:00Hope you enjoy the show. This is the year of software. More software getting built. More software developers coming in. More people building more software. More, more, more. I don't know about you, but my home lab is super active. My Proxmox is basically on fire over there. It's warm in my office because my stuff is always going so hot. It's kind of fun. I want to hear from you, though. What's your home lab like? Hanging out at my new fun hangout, howitworks.club. That's a fun place to be. If you're not there, you're wrong. You should check it out, howitworks.club. And, of course, hang out in Zulip, changelaw.com slash community.

2:01:35It's free to join. I'm there. You're there. Well, you're going to be there. I mean, go to that URL and sign up. It's totally free. And hang out in our Homelab channel and talk Homelab tech with us because that's fun. But we have some awesome sponsors, Depot.dev, our friends over at Tigard Data, and our friends over at Notion, Depot.dev, Tigard Data.com, and Notion.com slash changelog. Those are fun URLs to go to. Check them out. They love us. They support us. And of course, to our friends, our partners, our hosts, where we host our sprites, our machines, our apps, our everything. Fly.io. If you're not using Fly, well, that's just sad, man.

2:02:16Again, Fly.io. And to the beat freak in residence, Breakmaster Cylinder, the banging beats keep flowing. Love those beats, and I love Breakmaster. All right, friends, the show's done. We'll see you next week.

2:02:43Change log plus plus. It's better.