AI Agents Are About to Move Off the Cloud. Here’s What Changes

0:00Yeah, it's nothing but a piece of software, right? And all this, you know, software and all that, it has to follow the physical rules. And there are a lot of hard rules and the government said, if you can break down the task to a manageable size of task and have very clear instructions, they can do it. So I think that's fundamentally the philosophy of today's hybrid AI concept. I make mistakes and agents sometimes lie. Not sometimes, lie all the time. For example, the execution plan is laid out. This agent has to call these tools, go search that information and whatsoever. And then report it. I'm done.

0:48Success. But it never caught that tool. It never caught that tool. But it just lies right at your face. Success. Welcome, humans, to the Neuron AI Explained. I'm Corey Knowles, and I'm joined today by Grant Harvey, as always. Grant, how are you, man?

1:06Corey Noles:I'm doing good, Corey. How about you? Oh, I'm doing good. Excited to have this little chat today. Can you tell us a little about who and what we're going to discuss? Yeah. So today we are talking about where AI actually lives, because most of us experience advanced AI through enormous cloud models. But a growing number of PCs and devices can run smaller models locally, And Intel believes that the next phase will be hybrid, so an intelligent system that decides which tasks belong on your device and which require more powerful infrastructure. And for the record, I also agree with this. Today, we are joined by Dr.

1:44Corey Noles:Olena Zhu, who leads AI solutions and ecosystems for Intel's client computing group and has helped shape that strategy, including Intel's newly released SuperClaw. Before we get started, please take just a quick second to like today's video and subscribe above so you never miss an interview or one of our live streams. And on that note, Dr. Zhu, welcome to The Neuron. Thank you so much for the invitation, and I'm pretty honored. Thank you. Excellent. Likewise, likewise. Well, if you could, why don't you start with just giving us a little background on yourself and how you got into doing what you're doing at Intel.

2:24Yeah, sounds great. So I grew up in China. So in the early days, I had been already obsessed with math. I think that's how everything started. it. And so my PhD is also heavy math. It's on the computational science side. So when I joined Intel, I started with, you know, machine learning for design, basically design all kinds of algorithms to accelerate the, you know, silicon platform designs. And in recent years, we started to broaden up and to utilize machine learning and AI and the latest ILP and all this for day-to-day work and productivities and across all kinds of different professionals and disciplines, basically trying to scale the gen AI usages for Intel devices and platforms.

3:35And also for, you know, across industry, we really want to champion for local AI, hybrid AI usages for much more economical and much more private and safer and sustainable AI future. Wow.

3:55Corey Noles:Much needed. Yeah. It's for real. Could you talk us through a little bit what's reaching its limit in the cloud-first model of AI? And why do you feel like hybrid computing is the most practical path forward? Yeah, that's a great question. AI is nothing but a piece of software, right? And all this software and all that, it has to follow the physical rules. and there are a lot of hard rules and the government's it. You know, number one is the economically and from environmentally, right? So we have spent so much tokens in every company, every individual. We've burned so much tokens. It's not just coder.

4:46Corey Noles:Even just Grant and I. Oh, we did an assessment of our company's chat2bt usage And I think mine was like 800 million tokens for the month. It was pretty terrible. Yeah, yeah. You know, even if you just ask AI to write you a deep research report on some topics, it can easily eat up your entire subscription, right? Yeah. Within a few hours. So it's really, really hard. It's hard to scale and to sustain for every company. And another aspect is the data privacy, security. You know, on the enterprise side, it's very easy to understand, right? It's a lot of corporate secret data and customer information on that.

5:43You cannot just see they throw it into cloud and where does it go? Nobody knows. So I know a lot of the middle layer kind of agent companies and they said we have zero data policy, meaning, you know, but it still goes somewhere, right? So that's why a lot of government policies, regulations in Europe and a lot of places, and it makes AI adoption into different verticals become really hard. And even on the personal side, right, individual-wise, you know, there are so many, you know, fake videos and all that. I'm so concerned. I don't want to throw my daughters, you know, any pictures, videos, and to do any of those AI alternations and things like that.

6:44Yeah, right. Yeah, it's concerning. It's really concerning. And then on the other side is look at if we all centralize this AI infrastructure in data center and how many data centers do we need? and the water cooling and all these infrastructures and how much impact even to our day-to-day life. And I saw some articles and they are doing some in-depth assessment of this massive data center impacting the neighborhood power grid and even impacting every single appliance lifetime. Wow. you know, Yara household and also the, you know, like I said, the water pollution, all that, it's not sustainable.

7:40Another interesting aspect is for all these big companies, you know, model makers and all that, they also struggle and they want to, you know, push their services to all kinds of users, enterprises. But they are bottlenecked on the infrastructure. They couldn't scale. Right.

8:03Corey Noles:I mean, just look at Anthropic and how they've had to deal with their troubled rollouts. And, you know, all of a sudden you have Fable and then you don't have Fable. Suddenly growth is the biggest problem you have. Yeah. You have usage limits where you can't access it right when you're getting to the point where you actually need it. Then you run out for the five hour or the weekly limit. It's like it's very difficult to navigate for everyone involved. Yeah, exactly. That's why a lot of this, you know, model vendors and all that, they start to apply, you know, read limits and the pickle limit, all kinds of things.

8:39Yeah, yeah.

8:40Corey Noles:Oh, yeah. If you want to use it for anything creative with video, it's so prohibitively expensive. It is. Yeah, it is. And I think you hit on the head of something here that we've chatted about a little bit before, and that's that there really is a lot of incentive for companies to find ways to make this more efficient. Yeah. Absolutely. Building data centers isn't cheap business. Like, I'm certain that if it could be done with less, that would be the preference. Yeah, because then it makes the value of the data centers you do have that much more valuable and you can serve that many more customers too.

9:17Corey Noles:So there is certainly efficiencies of scale. Very interesting. What's up, humans? Managing virtual desktops across Microsoft environments can get complicated fast. Nerdio Manager for Enterprises brings Azure Virtual Desktop, Windows 365, and Intune together into one console. so IT teams can manage everything without bouncing between portals or stitching together point solutions. Nerdio also helps cut cloud waste automatically. Its intelligent autoscaling adjusts compute based on real usage, rightsizes machines, and can pre-scale environments before users ever log in, so you're not paying for capacity that you don't need.

9:57Corey Noles:And don't forget, security and compliance are built in, with automated policy enforcement and real-time threat monitoring included as standard. And for organizations that are moving away from Citrix or other legacy VDI platforms, Nerdio automates much of the migration process to reduce manual work and downtime. More than 15 ,000 customers already use Nerdio, from midsize to large enterprises. So get a demo of Nerdio Manager for Enterprise today at getnerdio.com slash the neuron. And now back to our show. So we've heard like versions of the mobile or edge versus cloud debate for years and years and years.

10:36What do you think has changed with generative and agentic AI? Do you feel like there's a sense of urgency now that didn't exist before? Yeah, I think everything is about timing, right? If you look at the history, everything is about timing, every technology and all that. So a lot of the technology like internet, you know, and the web, all that was invented started, you know, from cloud. But, you know, gradually a lot of the modules like Java engines and all that started to, you know, running or partially running local for a lot of reasons. You know, performance, privacy, latency and, you know, and a cost reason.

11:26So I think gradually it has to be a hybrid architecture and, you know, it has to be partitioned in a reasonable way. Some of these PCs and running on cloud, some of these PCs running on edge. Edge by edge, I mean all kinds of AI devices. And now it's a booming or about the booming of, I call it AI appliance. It's not only, you know, it's just like the form factor is changing. And while I was in China, I saw a lot of interesting stuff, AI metrics. And so it would talk to you and, you know, bend the higher ways you want to bend. It's basically embedding AI into all kinds of devices and the computer and all that will, you know, will all evolve.

12:24So the intelligence has to be distributed, like we just talked about, and sit where it needs to be. So it cannot be all centralized in one place. I agree.

12:42Corey Noles:Yeah, it's just not sustainable even as much as, you know, we would want it to be simple. It's not sustainable to do it that way. And also you mentioned AI devices. Well, imagine if every device in your house was sending everything it hears over the cloud. You probably wouldn't trust that very much. So I think some introspective consumer would probably think, well, you know, I want some of my devices if I'm going to let them into my home to stay on the device. And I think that's one reason why we want edge devices. Yeah. And also there's another perspective to this is the model's capability. Just in recent, you know, a few days or weeks, there was a very, very popular chart on Reddit.

13:35There's a hardcore community called Lokolama. Yeah, you guys know this, don't you? We know. Oh, yeah. Yeah, yeah. So there's a famous chart. And what it says is it really shows from, you know, how long does a model evolves, you know, from a frontier size of frontier model and to become a smaller size of model that can run on laptop. So the analysis started from GPT-3 generation, and it took 37 months. This model, the capability, the level of capability of this model actually came from cloud down to a laptop. And then analysis is every generation, you know, GPT-4, cloud is 3.5, and GPT-5 and all that.

14:32So average, it's about 24.8 months, you know, from a frontier running in about two, a small size model running on a consumer laptop.

14:48Corey Noles:Yeah. Consumer laptop. Do you think that that capability, because, you know, it started at 37 months and then now it's to 24, if I'm understanding the chart correctly. Is that timeline going to shrink even further then? Do you think? Yeah, so, and just based on this popular discussion, right, and what it says, if trends halt, then Faybo missile level classes of, the class of model capabilities could be running on this high device at the consumer hardware probably within two years. Maybe it could be shorter. Yeah, it's pretty wild, right? It's in two years. And it's like next year, this time, we can run quality for a lot of, you know, capabilities on a high-end laptop.

15:41Right.

15:42Corey Noles:Yeah. Wow. Yeah. That's something important. Something I'm wondering is that, like, you know, local AI is always sold through this kind of the privacy pitch is the idea. Like your data never leaves your device. But at the same time right now, local models are generally less capable than frontier cloud models. And that may continue to be the case for a while, at least for some amount of time, maybe, depending on the news of the week. Do you think that's a tradeoff users are willing to make, a certain amount of intelligence for a certain amount of privacy and security? Yeah, yeah, absolutely. It's a great question.

16:25So, you know, if you're running everything local and you get all these benefits, you know, you control your data and privacy and all that. But like you said, even model has been progressing really fast. But today running on local devices, we still see a delta between local and frontier models, right? But a user, you know, why we use AI is we want AI to work for us. And so the quality is important. You know, like I say, I'm not counting on my nine years old daughter to do some serious stuff because the quality is not there. But AI is like this, right? Different size of models is kind of like, you know, different similarities.

17:16And the smaller model is more like a middle schooler and the frontier is like a college student type of thing. Yeah, right. Yeah. So but then how do we accelerate like a middle schooler to put it, put this to, you know, this AI to work and still deliver reasonable quality? And that's how we actually pair them together. They need mentors. They need supervisors. They need certain ways and distribute the right task. You know, high schoolers could be really capable, right? Really capable. If you can break down the task to a manageable size of task and have very clear instructions, they can do it. So I think that's fundamentally the philosophy of today's hybrid AI concept, you know, and a product we are rolling out is we do a, you know, we do this routing or orchestration layer and really to look at for these specific tasks.

18:24Is this simple enough or is this very complex? And then do we need leverage frontier models and to do some decompositions? And then we allocate the right pieces to the local and to the cloud and then combine them together. So there are a lot of aspects to it. So the first one is cost saving. So let's assume we are doing some coding work and there's nothing super confidential. But still, if you're running everything on cloud, and it will cost a lot, that's why you have to offload some simple task to local. and another aspect is if the work involves a lot of confidential data, then how do we design a way to ensure a confidential communication?

19:30It's like the supervisor teach the small models and to look for the right information and put some structures to it, but, and the local model will never disclose any real data and local information to cloud. So those are the, you know, different considerations and the design aspect to this high-risk version.

19:58Corey Noles:I feel like I'm already doing a version of this where I'm using, you know, Fable as my orchestrator. And then I am having it, like, once it's figured out exactly what the task needs to be, In a coding example, like to follow your example, I'm then having it offload the work to a lesser model or in some cases a cheaper model like Codex, which is not that much cheaper, right? It's still frontier level, but I'm sending it off to Codex over the cloud and it's having it do some of the work to save some of my fable budget. Because as we talked about earlier, you have a limited usage that you have for the top tier models.

20:37Corey Noles:And so sometimes I'll be sending it off that way. But I know people who do that with local models. And I think that that is a pattern that's very repeatable. And I think that that should be what we should go for. It's like, yeah, you have the top tier intelligence that you can use and call on demand. And then you can farm it out to smaller models or preferably local models that can do some of the stuff on your device. Yes, and we saw some use cases that people even see after the task being decomposed, basically. And a lot of this repetitive work, it's better to be run on local because you don't worry about the tokens and all that.

21:27And then you can run more experiments and should do a deep optimization to kind of scan through all kinds of different possibilities. So we saw a lot of user comments around that. Nice. I'm kind of wondering what in the hybrid setup like you're talking about, what things like specifically make the decision of whether this should be handled in the cloud or down here? Because I assume this is happening in real time at some level.

22:00Corey Noles:Who's the traffic controller? Yeah, who's the air traffic controller and like what sets its, you know, its rules, constitution of sorts to know what goes where and make sure everything's in the right, you know, runway. Yeah, yeah. I love the question. Yes. So that's the OX Trader or router, you know, that's the design. so across the industry there are a lot of efforts actually doing this research model routing you know like Grant just mentioned across even multiple frontier models which one do we use for certain tasks that's on the frontier side and on the hybrid side is which one goes to the cloud model you choose or which one goes to the local model you use and all that.

22:56So routing is a very big research area, very active. So currently, the approach we are taking is to really, in a way, is doing a classification, meaning we look at this task and classify it into this is a simpler task, this local model can really handle, and this is a much more complex model, and we need the bigger model on-prem or cloud model to handle. But it's, you know, just in principle, it's like that. But a lot of details, a lot of aspects to consider, for example, like, you know, the confidentiality. So if this query involves a lot of confidential information, then where do you route? And do you do PII information reduction and go to cloud or not.

23:54So those kind of considerations. And so another aspect of this routing is, especially for local, because the local could have a lot of multitask, co-current tasks going on. Then does it have currently at this moment have enough resources to support this, right? So a lot of this kind of consideration are baked in and real-time sensing all these changing factors and making the decisions. Right. Okay. So suppose you have this workflow built and items chained together. What happens when something breaks? There's a kink in the process. Is there a – it feels like this essentially chain and nest. If something breaks, it could be a bigger problem than it would be when you're dealing direct with just one model.

24:54Yeah, actually, what you're touching is a fundamental problem in AI. You know, like AI make mistakes and agents sometimes lie. Not sometimes, lie all the time. I had one being lazy this morning I kept arguing with. Exactly. and he's like, for example, the execution plan is laid out. This agent has to call these tools, go search that information whatsoever, and then report it. I'm done. Success. But it never caught that tool. It never caught that tool. It never did it. Yeah, it never got it, but it just lied back right at your face. Success. That was my conversation this morning. I was having it assess our previous videos.

25:49And it's like, okay, there are 644 of them left to go. I'm going to start. And I come back and it's like they're all done. And I looked and it's traces and it's like looked at 32 videos. And I was like, hey, you didn't either. And it's like, no, I sure didn't. You're right.

26:09Exactly. Exactly. Exactly. It happens all the time. I mean, that's why, you know, like across industries, a lot of efforts going on with how to harness the agent harness, put the process, you know, audit and looping engineering. A lot of work and to make sure agent is being monitored and being traced and doing the right things there. It's always, you know, this trajectory can be, you know, can be logged and found and all that. So it's the same thing for the orchestrator or router design. And we have to make sure everything is being properly designed and, you know, logged, audited. And there's always a retry, fallback.

27:01So make the whole solution. It has to be robust, right? Yeah.

27:06Corey Noles:So talk to us a little bit more about how you've been implementing this with SuperClaw. Am I right to assume that SuperClaw is built off of OpenClaw or is a parallel to OpenClaw? How do they relate to each other? Yeah, actually, so SuperClaw is Intel's agent platform. and we aim to use it to help users, help our customers, and to really take advantage of AI agent workflows and also customize for themselves and utilize different modular IPs for their own designs. So that's the purpose of it. So how we designed it is we utilized open code as the framework. It's not open code. because we chose it carefully across all these community solutions because it's more towards the production qualities, active support and all that.