This solo builder runs 24/7 local AI on his own hardware | Alex Finn

0:00What is stacked around your office right now?

0:03Alex Finn:I have three Mac Studio 512 gigabytes. We got a DGX Spark as well as a computer. I just built an RTX 5090. Basically at all times of the day, each one of these computers is just burning tokens. The number one pushback I get on all this is your computers are so expensive. Isn't cloud models cheap? Isn't it$20 for a Chad GBT subscription? Well, that's not the point. The point is in pure ROI. The point is the use cases it unlocks. You now have, because you have AI models running locally, the ability to run unlimited intelligence around the clock 24-7. If you were to do that with a cloud model like ChadGBT or Claude, you would be spending outrageous amounts of money.

0:49What else fun are you doing with AI?

0:50Alex Finn:The most fun I've been having lately is building out my software factory. I have two loops in Claude code going. I have a build loop and a review loop. First, it has a build loop where it'll take all those tasks and start building out the tasks it came with over and over and over again. And then it has a review loop that takes all the tasks that were built and has another agent go in and review it. Once that's reviewed, it pings me on Slack and I can just leave a rocket emoji. And when I leave the rocket emoji, it says merged. And my Henry loop goes and merges it. It's been a blast kind of cracking this nut of how do you build your own software factory?

1:35Welcome back to How I AI. I'm Claire Vaux, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, I have Alex Finn, and he's going to walk us through his two Mac studios, DGX Spark and the NVIDIA-backed computer he built for himself and demystify what it means to run local models and have ambient AI working for you 24 hours a day. Let's get to it. This episode is brought to you by Runway, a new kind of creative platform that has everything you need to generate any image, video, or piece of content you want, all in one place. With Runway, it's now possible to go from initial idea to a finished deliverable in a matter of minutes.

2:19From turning low-fidelity product shots into campaign-ready imagery all the way through putting together big brand films, Runway can help your team scale your creative ambitions while keeping your budgets and timelines from doing the same. Runway brings together the world's most advanced AI models, which is why enterprises like Microsoft, Robinhood, Amazon, and Adobe, along with studios like Lionsgate and Legendary, all use Runway to ship real work every day. Try it yourself at RunwayML.com slash HowIAI. Promo code HowIAI. Alex, welcome to HowIAI Hardware Edition, Local Model Edition. I am so excited.

3:06So before we jump in, tell us just what is stacked around your office right now.

3:13Alex Finn:It feels like a sauna in this office right now, to be quite honest with you, because I got a lot going. I have three Mac Studio 512 gigabytes, which apparently puts me in the wealth class of Elon Musk having these now. I think they resell for like$30 ,000 each. We got a DGX Spark as well as a computer. I just built two days ago, basically a big computer built around an RTX 5090. So we have, I think, what's that come out to five or six different computers for AI. So a lot of hardware around here. And you are making really good use of it. Yeah. So I use it basically as what I call ambient AI to support my entire life.

3:58Alex Finn:And we'll go through all the things it's doing. But basically at all times of the day, each one of these computers is just burning tokens, doing things, helping my life. Right. Unlimited AI is basically how local AI works. And I all have them doing jobs around the clock for me, which you really can't do with cloud APIs. So how did we get here? How did you become hardware, local model guy? Like why is it so hot in your office? What brought you to this moment? it. My big awakening was back in January when I discovered OpenClaw. And so I was scrolling X one day on a Friday. I saw a blog post around OpenClaw.

4:41Alex Finn:I opened it. I'm like, wow, this is really interesting. Don't know why my gut instinct is like I got to buy a Mac mini. I go to this. This is before anyone was talking about OpenClaw, by the way. I get a Mac mini, put it on, start using it. It was one of the most aha awakening moments of my life. And something about building this like personal bond with the open claw and with this agent was like, I want this to live in the computer. I don't want this to come from the cloud. I don't want this. I want this working for me on my computer. And like, this is the future, just having this kind of personal assistant on your computer.

5:19Alex Finn:And so I started doing research. Okay, how can I run models locally? And I came inclusion. Oh, I'm going to go now after buying this Mac mini and buy a Mac studio with tons and tons of RAM and run local models and run it locally. And so that was kind of my, I guess, red pill moment into local AI was using open cloud. It only just advanced the last couple months. Then they started banning frontier models like Fable. And then the hardware prices start exploding and it just felt like everything was moving in this direction of like sovereign own your own intelligence and so i just keep investing more and more and more into it over the the last couple months so i i mean i had a very similar moment which is like just like got deeply claw pilled in in january like it just i remember i told this story i woke up one morning and turned to my husband because this is kind of things i say to him and i was like i'm having a chat like truly having a chat gpt moment where i think everything from here on out is going to be totally different so every red pill that we took is that little lobster color i think and i look i accidentally by accidentally i just like waited too long to get a studio and now literally i can't afford one or find one but i do have little fat stacks of minis back here that are doing Plenty of good work, but through cloud APIs.

6:43So I'm a little bit behind you.

6:46Alex Finn:I'm a strong believer, though, that those minis will be very useful soon enough. Like Google is doing a lot of research around like optimizing models so they can run on hardware like minis. So you won't be out of the game for long. Well, and I still find use for them. They're still happily at work. They're just at expensive, expensive cloud work. But let's go to, you know, you have all these machines. You have bought them and you have built them. Kind of what's good for what? Can you walk me through like how you make decisions and how you actually just get these models set up and running on these machines?

7:22Alex Finn:Yeah, so I've been experimenting the last few months around the different machines, the different capabilities, what they're good for, what they're weak for. Obviously, I started out with the Max Studio. I bought three of these 512 gigabytes. They've been great. But since then, I've experimented with kind of AI-focused computers like the DGX Spark. And then just recently, most recently, I've been buying kind of traditional GPUs from NVIDIA, like the 5090, soon the 6000 Pro. I put together a little chart here, which I'll show you. So you basically have four different options. Your Mac studios, your AI computers like the DGX Spark, which is getting very popular now.

8:06Alex Finn:Your kind of powerhouse, traditional NVIDIA chips like the 5090, building computers around these chips. And then basically everything else. Laptops, Mac minis, whatever you got sitting around your house. They all have different strengths and weaknesses, different reasons why you'd want to go for one over the other. First, there's the Mac studio. what's very powerful about Macs and the reason why a lot of people are going for them is they have what's called high unified memory. When you buy like an Nvidia chip, for instance, and you build it into your computer, you have kind of two separate types of memory.

8:45Alex Finn:You have your memory that runs all your programs. And then you have your memory, which is like your VRAM, which is for your graphics. And when you run AI models on that, it's all in the VRAM. The issue is the VRAM is very small. With the Mac, it's unified. So everything's the same. So whatever memory you have on your computer can be used for graphical processing. And so if you buy a Mac with a ton of memory, 512 gigabytes, 256, even 128, you can use all of that for graphical processing. And that graphical processing is what, you know, runs the AI models. So Mac studios are fantastic for huge, big models because you can use unified memory, right?

9:27Alex Finn:I'm running GLM 5.2, which is Opus 4.8 level, you know, intelligence on one Mac studio right now, which is unbelievable. The downside is you have very low memory bandwidth with Mac computers, which means it can't process a lot of it all at once, which means speeds are very slow, right? So the models are very, very slow. If I send a prompt to GLM52 right now on my Mac Studio, it might take five minutes for me to get a response back. So it is very, very slow. But you have frontier intelligence roaming on your hardware, which is amazing. Then you have AI computers. This is like the DGX Spark, very popular right now.

10:11Alex Finn:I think the price just went up to$4 ,600. Micro Center, I think, for$4 ,000. These are plug-and-play AI workstations. They have actually unified memory with NVIDIA, so you do get a lot of memory, 128 gigabytes, and you also get pretty decent bandwidth, and you get the NVIDIA architecture, which is called CUDA, which basically gives you a lot of speed as well. So it's kind of this sweet spot where you get decent memory and decent speed. And so you can run kind of these mid-sized good models like QEN 3.6, pretty quickly, which is nice. It's also very plug and play, these AI computers. You don't even need a monitor.

10:52Alex Finn:You just plug it into a wall and you can connect to it. And then lastly, you have your traditional NVIDIA chips. This is when you go on Twitter and you see the people saying, buy GPUs, and they have these huge racks of all these chips. These are all NVIDIA chips. And these are probably the most powerful of all the three. you get kind of lower VRAM, right? The 5090, which is a$4 ,000 chip, only has 32 gigs of VRAM, but it is lightning fast. It is extremely high bandwidth and you're getting like cloud speeds, but locally, which is really amazing. So these are really like your three. And then you have everything else, the Mac minis, your old laptop.

11:36Alex Finn:You can still run local models on them. There's options out there like Gemma 4, like a few other really small ones. They're not going to be anything close to Frontier, but you can do small things like embedding, which is basically managing memory for your agents. So you have these options. And so Mac Studio, just like big beefy models, high intelligence, slow kind of these AI computers, kind of like sweet middle spot. and then these chips very very speedy um and then you know dusty old computers maybe these like um point solution models that are like small for a specific use case that might be beneficial to be running in some context but are probably not going to be the thing that you just rely on over and over again um though you still on dusty old computers what i say is like you can still put them to work in terms of parallelizing your your cloud work across cross computers so we have a bunch of old computers and it's like I cannot do enough on my MacBook Pro I'm kicking off stuff to other computers just to like run work trees and do all sorts of interesting things I'm sure I could do that in the cloud too but you know I like to make use of all this this hardware sitting around no and it's it's uh I like to do that as well I have two Mac minis and I do the exact same thing.

12:54Alex Finn:And these apps are so good now, like Codex, for instance, is maybe the best agent harness out there right now. You can just go to Codex and be like, hey, go on the Mac Mini, right? Start building an app on there, test it yourself, do play right so you can go through the flow, screenshot things, send me videos. So I actually like running it locally, like on the Mac Mini as opposed to cloud, because you can have it actually click through and do things and then like then do screenshots of what it's doing. So it's it's a lot better for that, too. Completely agree. OK. And then how do you get these, you know, just high level set up with these models?

13:31What is kind of your typical install on any of these machines?

13:35Alex Finn:So the good news is OpenClaw and Hermes has made this process 10 trillion times easier, right before you would have to go find the right model, find the right version of it, make sure it can fit into memory, download it, run it on a server, all these really complex things that a normal person would never be able to do in like a thousand years. Well, the good news is you can now take Hermes or OpenClaw, basically make it your IT guy, say, hey, check out my Mac Studio, check out my DGX Spark, whatever you got, see what the hardware is, and then find whatever model you think is most appropriate for that hardware and load it up.

14:18Alex Finn:And then so as long as you have an agent, Hermes or OpenClaw, as well as TailScale, which basically allows you to create a private network across all your devices, your agents, basically your IT guy and can go across all of your devices, install whatever models and needs, set it up, do whatever you need. I'm going to show you a dashboard soon, which shows all my models working and running. It's all coordinated and run by my Hermes and my OpenClaw. So as long as you have Hermes OpenClaw going, you need to say, hey, OpenClaw, check out the new Mac Studio I just bought. Look at all the hardware, figure out which model we should run.

14:57Alex Finn:Think about the use cases that are appropriate for me and then load up a model and get it going. And like you don't need any technical knowledge whatsoever. They'll go across your devices on tail scale and set it up for you. There really is like no technical work needed. And so to repeat this for folks, just so I'm understanding, you have a machine set up with OpenClaw. You also have all your machines networked on Tailscale, which allows you to have this like little virtual private network. And then that one sort of like IT guy, OpenClaw, or Hermes agent, can then use Tailscale to just go into these different machines, configure and manage them.

15:32Alex Finn:Yeah, exactly. So tail scale is worth getting even if you just have one computer because it creates this private network where even if you're vibe coding like on your MacBook Pro and you just have a phone, you can like go on the local host on your phone and from that's on your computer and like test your local apps out because it's all in the same network now. But it's even better if you have multiple computers. So maybe you have your MacBook Pro, then you buy a Mac Studio for AI or a DGX Spark. You install Tailscale and all of them. And then you can just say, hey, go on this other computer and do this.

16:06Alex Finn:Load up the model, whatever. And it will jump between all your devices. No technical knowledge needed. And load up and run anything you want. This episode is brought to you by Jira Product Discovery. AI has made individual PMs incredibly productive. But multiplayer mode is where it still breaks, getting everyone aligned on what should actually get built. Decisions live in a markdown file from last week. The roadmap's a spreadsheet no one's looking at. Jira Product Discovery is where teams actually decide what to build. Capture ideas, prioritize them as a team, and share a living roadmap everyone works from.

16:43It's powered by Atlassian's Teamwork Graph, so it can pull in customer feedback, what your team shipped plus your goals and suggest what to build next. And when a decision is made, you can hand it off straight to Jira so a developer or even an agent can pick it up and start building. Teams at Canva, Deliveroo, and Toast already use Jira Product Discovery. Join more than 25 ,000 teams at atlassian.com slash howiai. Start building the right things together. Okay, so we've talked enough about hardware. We've talked enough about how to get it set up. Show me how you use it. So what are you using all this sort of intelligence for?

17:27And how do you keep it burning tokens, you know, effectively?

17:34Alex Finn:I've been building a system over the last couple months to coordinate all these computers and all these models. I do a lot of different things just to kind of set the stage here. The number one pushback I get on all this is, well, your computers are so expensive. Isn't cloud models cheap? Isn't it$20 for a Chad GBT subscription? Why the hell would I buy a$10 ,000 Mac Studio? That's like 11 years of Chad GBT usage. Well, that's not the point. The point is in pure ROI. Not everything in your life is pure ROI dollars and cents, right? The point is the use cases it unlocks, right? You now have, because you have AI models running locally, the ability to run unlimited intelligence around the clock 24-7.

18:26Alex Finn:If you were to do that with a cloud model like ChadGBT or Claude, you would be spending outrageous amounts of money. So you wouldn't be running it 24-7 burning tokens around the clock. But because it's local, because you have unlimited usage of it, you can burn tokens around the clock. And so that brings me to my fleet control, I call it. This is my fleet dashboard, which allows me to see all my computers, which models are running on them currently, and monitor everything they're doing and organize their 24-7, 365 tasks. Throughout the day, the local models I have running are constantly doing work for me to support my life, to support my many lines of business.

19:13Alex Finn:I'm building a SaaS right now, Henry Intelligent Machines. Every 30 minutes to an hour, one of these local models does a security scan. So actually picks out an API endpoint or some part of my code, runs a security scan on it, and makes sure it's secure. Another local model, every half hour or so, does a code review, picks out some piece of code, and de-slopifies it, right? Finds ways to optimize it, finds ways to speed it up, finds ways to make it better. Another local model will, every 20 minutes, look at Twitter, Reddit, Product Hunt, Hacker News, and look for Signal. As a problem solver, the only way I can solve problems is if I find them.

19:59Alex Finn:Right. And so I have this ambient AI going online 24 hours a day, reading all the social media sites, looking for signal, looking for challenges people are having. If someone goes, man, I really wish I had a piece of software that allows me to edit my videos like this or something like that. My agent will find that signal, put it in my queue and I'll be like, OK, can I build a SaaS to solve this? Can I build a program to solve this? And there's many screens here I can go through, but that is from a high level. What these agents are doing and the strength of local AIs, you can have it running 24-7, 365, just doing different things online for you.

20:40And so for that, let's just talk about the coding use cases really quickly. So for these like automated security scans, automated like quality checks, do you feel like the local models are of sufficient intelligence to get the job done? And are there specific models that you've applied to, in particular, the coding use case?

20:59Alex Finn:When getting into local models, you really want to understand what is the delineation between what you want to do with local models and what you want to do with frontier intelligence. intelligence, right? You don't want your entire security apparatus to be run by local models. The intelligence just isn't there yet. But where the advantages and the strengths come in is basically it runs this security scan, looks for these challenges. And then what it does is every day it builds a report. And you can see some of them here where it'll build a report around what the security issue is, what the code snippet is, what the problem is, put it all in this markdown file that describes like, for instance, today's has 374 findings in it of security issues.

21:46Alex Finn:Every day I have a loop running in Claude code, so slash loop 24 hours, where it goes and takes whatever the latest finding is from the local models and reviews it and then goes through the code exactly where it points to and sees if it's a real issue and how to fix it or not. So it's, you know, if I were to loop Claude code, like the local models doing every 20 minutes, look at different, I'd be spending thousands, thousands, thousands of dollars a month. But the, you know, the, this is almost like the business development rep, right? That's going qualifying leads and then Claude codes, like the clothes are taking those leads and go, okay, what can we fix?

22:24Alex Finn:So like you don't have the clothes or doing all the work. How do you federate work across these machines? And so we're like, do you have some that you're like, this is my coding machine. This is my market research machine. Are you like round robin? Again, we're like using very SDR terms. I'm like, are we round robin these leads? Like we've both been in the SaaS world way too long. Exactly. Yeah. How do you how do you do that allocation? Again, every model computer has different strengths. GLM-52 is opus level, right? But it's very, very slow. So I don't need it doing super fast work. So I can have it doing the security scans, right?

23:08Alex Finn:I can have it go and do security scans. Even if it takes 24 hours, that's fine. Claude Coats only check in the report once a day. So it can do the kind of super critical, smart stuff. On the other end, Quen-36, which is not quite as smart as GLM, I can have that reading Twitter and read it all day just looking for signal. That's a very simple thing to find. Look for someone who has a challenge. And so it's just pulling in data, crunching it, and reading for challenges, which you don't need the highest intellect. So you just need to know, like, what's the strength of the computer? What's the strength of the model?

23:40Alex Finn:And what would be tasks that be appropriate for those strengths? Is it in this daily brief that you have these tasks assigned to you? Like, is this the dashboard for you, the agents and the machines all to collaborate? So I'm going to advance it to the point where it's like collaborative with Claude Code and Codex eventually. Right now, the system where Claude Code is like looping every day and going in, that's happening separately. It's just in its own chat inside Claude Code where it loops over and over again. But no, this is all around my local models. But I think where the connector is with everything else going on is with my OpenClaw and my Hermes.

24:19Alex Finn:They're basically the guy that runs this as well. this dashboard then communicates with Claude Code. Hey, can you kick this off? Can you do this? So it's kind of all glued together by my personal agent. Okay. I have to ask you a question. Open Claw, Hermes. You run both. You have a favorite. Tell me. I use both because much like Claude Code and Codex, it feels like there's some days where one is really, really dumb and the other is significantly smarter. And I feel like uninstalling one and getting rid of the other. I'll say this, though. The dependability of Open Claw turned me off. There was a run for like a month straight where every update I did broke it, and then I had to spend half an hour fixing it.