Local AI Clearly Explained

0:00I think local AI and open models are going to create a ridiculous number of business opportunities over the next 24 months. And I don't think most people actually have the map yet. They've used ChatGPT, they've used Claude, but when they hear local AI, HuggingFace, Ollama, LMStudio, AI Edge, it sounds like it's for this developer world and that normal founders are just not supposed to touch it. And I think that's a mistake because the opportunity here is actually pretty endless. By the end of today's episode, you're going to understand what local AI is, when it matters, how to run open models that work, where Hugging Face fits in here, which Gemma model I'd start with, how I'd run a model locally with LM Studio or Olama, and how this turns into real business ideas.

0:46And I'll give you three startup ideas I'd actually consider building using local AI, including who the customer is, what the first version does, why local matters, and how I'd sell it. But basically, this is going to be a masterclass around local AI, how to run models, how to build apps, how to make money from it. And I'm going to explain it for the average person who isn't technical. Quick shout out to Google for sponsoring today's episode and for caring about local AI and open models for entrepreneurs. Today's episode, I'm going to use Gemma and Google AI Edge as the main examples. But the goal is to give you a full map so you can actually understand the space and build with it and use whatever model suits you.

1:25Okay, let's dive in.

1:35So put simply, local AI means the model runs on hardware you control. The hardware could be your MacBook, it could be your Windows laptop, an Android phone, an iPhone. It could be a browser, a Raspberry Pi. It could be in a workstation in your office. I just got a DGX Spark, which is like a high-end one. But the important part to note is a phone now could run local AI. Cloud AI means the model runs somewhere else and you access it through a website or an API. And that's the basic difference. The business question is, where should the intelligence live? If I'm doing deep research and strategy and hard reasoning or something where I want the strongest possible model, I'm probably going to be using a frontier cloud model.

2:23If the work involves private files like sensitive customer data, offline usage, fieldwork, low latency, audio input, or an internal workflow that runs again and again and again, local AI starts to make a lot of sense. A smaller model in the right place can actually be very valuable. That is the idea I want you to keep in your head. The first question most people ask is, is this model smarter than the biggest model in the cloud? The actual more useful question to ask actually is, is this model good enough for the job? And does running it locally make the product better? Once you ask it that way, you start seeing these business opportunities, which we'll go into.

3:09So there's four pieces to the local AI landscape. the model which is the brain file that could be something like gemma llama or mistral the warehouse which is where you find the model you might have heard of hugging face i think they're trying to get acquired right now at 13 billion dollars um that's what they do the software which is what runs the model that's something like lm studio or olama and then the workflow which is the product you're building around all of it. And those are the real four pieces. The model is the brain file. Gemma is a model family. Llama is a model family. Quen or Mistral, you might have heard of Phi.

3:51These are model families too. Some of these are actually better at reasoning and some of them are better at coding. Some of them are smaller. Some of them are faster. Some of them are better for images. Some are easier to run on your own machine. And then you need somewhere to find these models. That's what Hugging Face is. That's the first place I would go. They're the biggest at it. The easiest way to explain Hugging Face is that it's a model warehouse. You go there and you can find model cards, licenses, file formats, examples, benchmarks, community versions, and sometimes versions that have already been compressed so they're way easier to run locally.

4:32If you're new to local AI, one of the best exercises is actually just to open hugging face and read a model card really slowly. You're going to learn a lot. At first, though, you can ignore half the scary looking details and just look for a few basic things, in my opinion. What is the model for? How big is it? What license does it use? What hardware are people running it on? Does it support text, images, audio, tool use, or embeddings? Are there quantized files available? Once you can answer those questions, the space gets a lot more intimidating. Because I know when I first looked at these cards, initially, I was like overwhelmed.

5:14So just those are the key questions to ask. Then you need software that runs the model. For most people, I would just start with LM Studio or Olama. LM Studio feels like a normal desktop app. You download it, you search for the model, you click download, and then you can just chat with it. My opinion is it's probably one of the most friendly first-time user experiences if you're non-technical. Ollama is a little more builder-oriented or developer-oriented. You install it, you run a command like olama run gemma 4 colon e4b, and now you have a model running locally with an API your apps could talk to.

5:56Then underneath those tools, you're going to start hearing about things like llama.cpp and mlx. And I'll explain what those two things are. Llama.cpp powers a lot of the local model inference. MLx matters if you're on Apple Silicon. And if you're thinking about shipping real on-device apps in the Google ecosystem, that's where Google AI Edge and Lite RT LM come in. Basically, Google AI Edge is the broader on-device AI development world, and Lite RTLM is the runtime layer for the language models. This is what you study when you want to move from, I ran a model on my laptop, to I want this model inside an iOS app, or an Android app, or a web app, desktop app, whatever it is.

6:48we got to talk about some key vocabulary just about the most important things you need to know about these words that uh that come up time and time again in local ai i'm just going to give you simple clear definitions what they are by the end of this part you'll know you know just the core basics of of local ai vocab so i'm sure you've heard this one before uh parameters like two billion four billion these are what's called the internal weights of the model and more parameters just usually means more capacity for harder tasks, but it does require more memory. So parameters are the internal weights of the model.

7:27The beginner shortcut is that more parameters usually means more capacity, and more capacity can help with the harder tasks. So the trade-off is usually memory, speed, and hardware. So a 2 billion or 4 billion model is the kind of thing you might use for edge devices, phones, fast workflows, and smaller tasks. A$12 billion parameter model is more of a middle ground. And a$26 billion or$31 billion model is getting to the stronger workstation territory, depending on your hardware and how the model is built. I recommend not going out there and spending$5 ,000,$10 ,000,$20 ,000 on a workstation just yet.

8:11By the end of this episode, you're going to understand how to just set up some of these things on your phone or on a laptop, a spare laptop that you have from 2021. Then there are tokens. So tokens are the chunks of text that the model reads and writes. Locally, you care about speed and memory rather than the per token bill. Then there is the context window. The context window is basically how much information the model can work with at once. Then there is quantization. The word sounds more technical than it needs to. Honestly, I can barely pronounce it. Quantization is the compression for models.

8:49It allows giant models to fit on normal laptops. For example, you might have heard of Q4, Q8 formats. That's quantization. If the full model is the giant version, the quantized model is the version that can actually fit on a normal laptop. So you might lose a little quality, but suddenly this thing magically runs. You will see things like Q4 or Q8. And as a beginner rule, Q4 is just usually easier to run. And Q8 keeps more quality, but it needs more memory. If you're just getting started, Q4 is just a reasonable place to begin. So I would start there. Then there is GGUF. It's a common file format for local models that make inference easier on normal machines like you and I have.

9:36And in the Google AI Edge world, you'll see something called the Light RT LM. This is the model format and runtime path you care about when building on-device apps with Light RTM. So the simple map is this. Hugging Face helps you find and understand models. Gemma is Google's open model family and Google's a trusted brand. I run my business on top of Google, so it just makes sense. LM Studio helps you try models locally without much friction. Olama helps you run models locally in a way that a bit more technical people can plug into apps. GGUF is a common local model format. And Google AI Edge and Lite RTLM are the path toward shipping on-device AI products.

10:28That's what you need to know. So let's talk about Google's open model family because I feel like there's a lot here. It's a bit overwhelming and I'm just going to break it down so you understand what you need to know about the whole Google AI open model family. So Gemma is Google's family of open models and Gemma 4 is built for the efficient local and on-device use. So they have Google 4 E2B, which is the smaller edge model for phone workflows. You have a bigger E4B, Gemma 4E4B, which is pretty much the most practical starting point for most local tests. Then you have Gemma 412B, which is a middle ground with more capability for laptops.

11:15And then you have Gemma 426B slash 31B, which is the stronger local workstation territory. That is the main model picker. Then, and a lot of people don't know this, there's specialized Gemma models that are just really useful to know. So you have things like embedding Gemma, which is just for search. So specifically, it helps you turn text into embeddings, which lets you search by meaning. If you want to search your own docs or customer notes or support tickets, sales calls, or knowledge-based localies, embeddings matter a lot. Then they have something called function Gemma, and that's a tool use in structured function calling.

11:56That means the model can help software take actions in a way more structured way. It is part of the path from the model gave me an answer to the model helped the product do the next step. Then you have a few more like Pali Gemma, which is more vision focused. You have Shield Gemma, which is more safety focused. Then you have Gemma Scope, which is more understanding how models work under the hood. You can leave most of the family alone on day one. The practical path, like on day one, if you're a beginner, start with Gemma 4E4B, understand the workflow. Then you can move up or down or sideways, actually, depending on what you are building.

12:38So the way I understand the whole Google AI ecosystem is you have Gemma as the open model family. You have Google AI Edge, which is the on-device AI development ecosystem. You have Lite RTLM, which is the runtime for running languages models across all the devices. You have AI Edge Gallery, which lets you try on-device models and see the experience just more directly. And if you need huge scale, things like strong managed infrastructure or frontier-level cloud reasoning, you still have Gemini and Google Cloud that you can use or another frontier LLM that you can use. The reality is a lot of big products and serious products are going to use a hybrid setup.

13:26They're going to use cloud for certain things and you're going to use local for other things. As an example, imagine a local AI tool for a professional service firm. So the local model is going to read the sensitive drafts, checking for the issues. It's going to strip or summarize all the private details and prepare a clean version of the problem. Then, when the customer wants deeper reasoning, a cloud model can help with the sanitized version. That, to me, feels like a more natural architecture than just putting everything into the cloud, which a lot of people don't want. You basically have local handling, the private files as a first pass.

14:08And then cloud handles the heavy thinking when you need it. A human can improve the work before anything important goes out. This is how I'm starting to think about building a lot of these products. Beyond Google Gemma, I'll give you a quick primer on the other families or other open model families you'll hear about and some of the pros and cons. Llama is a Meta's model family. And it's probably the default open model reference point for a lot of developers because it's a pretty big ecosystem. The upside is the community, the tooling, the examples, support. The downside is you still need to read the license and the model card, especially if you're building a serious commercial product.

14:54Quen is Alibaba's model family and has become very strong, especially around coding, multilingual work, long contacts, and agentic tasks. The China thing is real. A lot of people use Quen because it performs really well. But if you're in an enterprise, a government, healthcare, finance, or sensitive data environment, you need to separate running open weights locally from sending data to a hosted service. And you need to check what your company is comfortable with or what you're comfortable with. DeepSeek is similar in the sense that it's made a lot of people realize how strong Chinese-based open models really could be.

15:39especially for reasoning and coding. The upside is performance and cost. It's pretty cheap. The trade-off is that some buyers will have procurement, security, or geopolitical concerns. So I'd be thoughtful about where I'd use it, how I'd deploy it, even if you want to use it and go down that path. There's also GLM, or people know it as z.ai. It's another one you'll see pop up a lot. I actually did an episode on it, especially if you spend time on Hugging Face and Olama or just local model Twitter, you're going to see it a lot. The thing to know is that some of these models can be weirdly good for specific jobs.

16:22So I wouldn't ignore them just because they're not the obvious brand name. You can test them. You can read the model card. You can check the license and just play with them. But what you might deploy in the sense of for your business or for what you're doing might be very different. There's also Mistral, which is the European model family. I think they're based in France. If you care about efficient models, they do a lot of releasing, a lot of practical developer use cases. They're pretty good. It's a strong model with a pretty builder-friendly posture. But the downside is the lineup is a little confusing.

17:01Some models are open, some are commercial. so some question marks there. Microsoft also has their open model family. It's called Fi. I think it's interesting if you care about smaller, faster, lower latency models but for a lot of use cases I haven't seen it work very well. And honestly, there are new models showing up all the time. It feels like every other day and that's why hugging face matters. You're not going there just to find the big models. You're going to find these weird specialist models, these community fine-tunes, quantized versions of stuff, these forks, and model cards that tell you whether something's actually usable for the workflow.

17:51You don't need to memorize all of this, but the takeaway basically is that there's these ecosystems and your job as a founder or just someone who's playing with these models is to pick a model family that fits your workflow, that you connect with that company, you like how they do things, and then go from there. You can play with a lot, learn a lot, and then pick a family. So how do we make this whole thing real? If you actually want to run Gemma, here's how I would do it. I would start with LM Studio. I would download LM Studio. It's free to download. You open the app, you search for Gemma 4.

18:32If your machine is solid, try E4B. But if your machine is a bit slower or older, I would look for E2B. And then I would look for the quantized version if you're using the GGUF path because you want the model to run just a lot more comfortably. Once it downloads, open a chat and just ask it something really simple. I would use a business prompt because I want you to feel the value immediately. It's sort of an aha moment. Maybe it's something like, read these customer notes and turn them into a one-page memo about what customers are struggling with, what has changed, and what the business should fix this week.

19:16And then just paste some customer notes or just fake customer notes just if you want to see the value. The point of this exercise is just basic. the model is now running on your machine and you're using AI without sending that prompt to a cloud model I believe everyone should try that and feel what that is because I do think that it's just going to be a lot more common and it's going to unlock your brain in a completely new way after that, go to LM Studios developer section and start the local server because that just gets a lot more interesting because other apps can talk to the model on your laptop Your computer becomes this little AI server.

19:58So you can have a script or a prototype or just an internal tool that can call the model through local host and you get an answer back. I think that's when you start to see how products are going to get built in the modern age. The second path is Olama. So install Olama and run Olama poll Gemma 4. then run olama run gemma 4 colon e4b. Now you have gemma running locally from a command line. Olama also gives you a local API port. I think it's on 11434. It is useful because you can connect your own app or script to it. If you want to test a larger model later, you can try the 12 billion, 26 billion, or 31 billion versions, assuming your hardware can handle it.

20:59And you can ask an LLM if your hardware can handle it, or you can do it yourself and just suffer through the slowness and the pain of it. The third path is Google AI Edge and LiDAR TLM. I would only use this path if I wanted to build an actual app and a model inside of it. For example, maybe I'm building a mobile app and the model is running on the phone. Or it could be like a browser app where the model runs locally. Or maybe it's a desktop app with a private workflow. Or something on an edge device. Light RT LM is designed for that world. Android, iOS, web, desktop, and edge environments. That is the path from local AI as a demo to local AI as a product.

21:52So here's the hardware cheat sheet that I would use. If you have 8 gigabytes of RAM, start small and keep the first test simple. But if you have something like 16 gigabytes of RAM, you can do some useful experiments with models like E4B and smaller quantized models. If you have 32 gigabytes of RAM, you have way more room to do larger local workflows. If you have a strong GPU or a workstation like a DGX Spark, the bigger models become just much more realistic. And for phones, I would think a lot less about model size and more about the job. So can the model understand a photo? Can it summarize audio?

22:34Can it classify something quickly? can it help a worker in the field? Can it run without a strong connection? Can it do something useful inside the app before the user even thinks to ask? Now let's build the first workflow in our heads. So I would make a folder on your desktop called customer notes. And inside that folder, I'd put 10 support tickets for a specific business. Let's say it's a home health agency or a med spa. or a water damage restoration company. The notes might say something like, I tried to reschedule but couldn't find the link or the technician didn't explain what happens next or hey, no one actually confirmed my appointment or I was charged twice here.

23:21Then I would run a local model like Gemma and ask it to create a file called whatcustomersaretellingus.md, the markdown file. The output should include the repeated complaints the exact customer language, the likely root cause, the part of the business that seems broken, and the one thing the operator should test this week, the high-priority stuff. This is a good first local AI workflow because it's useful and it's simple. What do you have here, right? You have this private, messy data. The model runs next to it, and the output is a memo someone could actually use. And then once you actually go and, you know, you're going to go and do this and get the output, but you're going to, like the unlock I was talking before, it's going to unlock something in your brain.

24:11You're going to see this pattern everywhere. A folder of customer calls become a market research memo. A folder of support tickets become a product roadmap signal. A folder of PDFs become like a risk checklist. A folder of drafts become a pre-send reviewer. This is why I always start with workflows before I'm fine-tuning anything. people hear you know open model and immediately want to train their own model and i get it i get why i was actually the same way um it sounds really cool but i feel like that's like an advanced move the practical move the beginner move where you should start is just to find a repeated workflow first you pick one folder one model one output and you run it like 10 times you see where it gets confused.

24:59You see where you can improve the prompt and add examples. You add a checklist and then you create a small eval. What's an eval? An eval is just a small, it's just a test that tells you whether the model did the job well enough. For this workflow, for example, the eval could be really simple. It could be like, take the same 10 customer notes and run them through Gemma locally and then run them through a strong cloud model, a frontier model, and then just compare the outputs. And then you ask, did Gemma catch the same complaints? Did Gemma pull the right quotes? And did it follow the format? Did it miss something?

25:42The comparison actually teaches you where local is already useful and where you still want that stronger cloud model and how you should think about the hybrid model I was talking about. That's really how I think about local versus cloud decisions. Use local for private, repetitive, fast, offline, device native, and high volume workflow. Stuff that you want to run all the time. Use cloud for deep reasoning, giant context, broad research, and cases where the strongest model changes the quality of the answer. So you use both when the product has sensitive data and hard reasoning. A lot of valuable products will work that way.

26:25You know, local first pass, you do the cloud escalation, human approval for anything important. I think that's the way work's going to get done. So I want to give you three startup ideas where local AI actually matters. And these are the kind of businesses I would look for, niche, useful, cash flowing businesses that you don't need to raise venture for and tied to a painful workflow. The filter is pretty straightforward. So I look for a customer with sensitive data, repeated review work, bad software usually, expensive mistakes, mistakes that will cost them a lot, and a workflow that happens close to the device.

27:05That combination is the interesting zone for me. So let's go through the three ideas. I want you to steal these ideas. And at the very least, it'll get your creative juices flowing with how you can use local AI to run model, build apps and make money. Idea number one is a local QA reviewer for home health agencies. So home health agencies have nurses and caregivers and they go into people's homes and they write visit notes and updating care plans and dealing with billing and compliance. The paperwork is a pain. It takes a lot of time if you've ever witnessed it in person, but it matters so, so much.

27:45Like a missing detail can create a billing delay and a vague note can create extra admin work. And a mismatch between the visit and the care plan can create a ton of risk and we don't want that. So the first version is a local desktop app for the agency. The agency drops in visit notes and care plans and dictated transcripts. And then the model is going to review them before the submission and should look for flags. So it's going to flag things like this note mentions dizziness, but vitals are missing. Or the caregiver described a medication change, but the follow-up instructions is pretty unclear.

28:25Or the note may not support the billed service level. The buyer mostly cares about fewer documentation problems before the billing or the audit or a supervisor review. So if you solve that, you have their attention. Now, I don't want to just give you the idea. I mean, how would you actually grow this? If I was starting this business, how would I grow this business? I would actually start it as a service. So I would find five small home health agencies, and I would offer to review a batch of notes. I would do the review with AI, helping behind the scenes with the local AI. And I would inspect everything manually with human beings, myself first.