Actor-Critic without Actor: Critic-Guided Denoising for RL

0:00If you work in machine learning, you know, the default answer always seems to be just add another network, right? Make it bigger. Right. Throw more compute at the problem, standard procedure. But what if I told you that the secret to, like, drastically improving a complex system, specifically in reinforcement learning, was actually taking out half of its components? That's a pretty radical idea. It is. And we're diving into a potential paradigm shift today that really champions simplicity, efficiency over just massive architectural bloat. Yeah, and that's exactly what our source material digs into.

0:36We're looking at a research paper that introduces something called Actor-Critic Without Actor, or ACA for short. ACA, okay. It's a lightweight framework, and it fundamentally challenges that traditional two-network setup we see in deep RL. They're basically arguing that the actor network, you know, the part that picks the actions, it might be computationally redundant. Redundant, wow. Okay, so our mission today is to unpack that. Like, how is that possible? We need to start with the basics, the history. What are actor-critic methods? Right, the foundation. Why do they hit these computational, these learning roadblocks?

1:07And then, yeah, how does ACA manage to just ditch the actor and replace it with something that actually works, guided just by the critic? And the payoff for you if you're working in this space seems pretty huge. This approach, well, it simplifies the whole architecture, potentially improves stability during learning, and drastically cuts down the number of parameters you need. Which means less compute, less memory. Exactly. They show, for instance, in complex environments, ACA runs on about like 0.677 normalized parameters. Compare that to 1.000 for a standard method like soft actor critic or SAC.

1:43So almost a third less. Pretty much. That's real efficiency, real hardware savings. Okay, let's get into it. Before we talk about subtraction, let's define what we're taking away. The standard actor critic or AC paradigm. Yeah. It's been the bedrock for a lot of modern RL, especially continuous control. You've got two networks working together, right? Exactly. Two separate neural networks. You have the actor, usually written as pi, and its whole job is to look at the current state and pick the best action. The policy network. The policy network, yeah. And then you have the critic network, the Q function, which estimates the value of taking that specific action in that state.

2:17Like, what's the expected future reward? And it's this cycle, right? The critic evaluates, the actor improves its policy based on that evaluation. Right. Policy evaluation, policy improvement, back and forth. It's a foundational loop. Seems elegant enough on paper. Critic informs actor. Actor gets better. So where does the inefficiency come in beyond just, you know, needing two big networks? Well, that resource overhead, double the computation, double the memory. That's definitely problem number one, especially with large models. But I think the bigger conceptual issue is something called policy lag.

2:51Policy lag. Okay. Think about it like this. The critic, using methods like Q-learning, it can figure out the best state action pairs pretty much instantly. It sees the highest value, the best immediate move. The ideal target. Yes, the ideal target. But the actor network, it can't just jump straight to that target. It has to update its policy gradually. It uses policy gradient methods, which means small steps. So it's trying to catch up to what the critic already knows. Exactly. It's like driving using a GPS that gives you the perfect route, but the route only updates every five minutes. You're always following slightly old information.

3:25Ah, okay. The actor's always trailing behind the critic's most up-to-date knowledge. Right. And that lag means slower learning overall, slower convergence, and it can introduce instabilities. Okay, so the standard AC setup is complex, it's resource heavy, and it has this built-in delay, this policy lag that hinders things. Now, before ACA came along with its no actor idea, the field tried something else, right? Using diffusion models as policies. That's right. Diffusion models became, you know, the shiny new object for a while there. And for good reason in some ways. They offer incredible expressivity.

4:00Meaning they can represent more complex action patterns. Exactly. Traditional policies often just output a simple Gaussian distribution, like a bell curve around a single action that limits exploration. But diffusion policies, they can capture really diverse, complex, and importantly, multimodal action distributions. Multimodal meaning? There might be several different equally good ways to do something. Precisely. Think of a task where maybe turning left or jumping right are both good solutions. You want your agent to be able to explore and represent both possibilities. Diffusion models are great at that.

4:34Okay, sounds powerful. But as usual in deep learning, I'm guessing there was a catch, a cost to that expressivity. A big one. The complexity just exploded. To make diffusion policies work, you need these huge denoising networks. Denoising networks. Yeah. Because diffusion works by starting with noise and gradually removing it to get the final action. Exactly. And those denoising networks are monsters. They massively inflate the parameter count. They eat up memory. And they make training much slower. So we gained better exploration, maybe, but we ended up back in the same efficiency mess or even worse.

5:08Pretty much, yeah. More complex to implement, harder to scale. It didn't really solve the core efficiency problem for many applications. Okay, so that sets the stage beautifully for ACA. This is where it gets really interesting. ACA's big idea. Just get rid of the actor network entirely. So how on earth do you generate actions if there's no actor network telling you what to do? By making the critic pull double duty, essentially, it relies entirely on the critic's value judgment as the guide. They call it the critic-guided denoising process. Critic-guided denoising. Okay. So instead of training a separate network to output an action, ACA reformulates the action sampling process.

5:49It uses the gradient field of the critic itself. Hold on. Let's unpack that. The gradient field. You mean like the direction of steepest descent on the value landscape. Precisely. Think of the Q function, the critic, as mapping out the value of all possible actions in a given state. The gradient of that map points directly towards actions that promise higher value. So ACA uses that gradient information directly within the reverse diffusion process. Remember how diffusion removes noise step by step? Yeah. ACA guides each of those denoising steps using the critic's gradient. It's basically saying, at each step, adjust the noisy action slightly in the direction that the critic says leads to higher value.

6:27So the critic isn't just passively evaluating anymore, it's actively steering the action generation. Exactly. It creates what they call an implicit actor. Let's call it DuffKeyDollars. The policy arises directly from the Q function. And the key advantage there is? No more policy lag. Because the action is being derived directly from the critic's current up-to-date value estimates, there's no separate actor network that needs to slowly catch up. Ah, the alignment is immediate. Immediate. The mathematical formulation, it shows the noise prediction epsilon by is directly guided by the gradient of this special critic.

7:01Epsilon W sigma and lava, that tight coupling bypasses the whole slow policy optimization step. That seems huge. Eliminating policy lag instantly. Is that the biggest win here, practically speaking? It's massive, yeah. Think about how much training time in standard AC is spent just getting the actor to catch up to the critic. millions of interactions sometimes by deriving the action straight from the critics gradient you cut out that entire loop it should lead to much faster convergence and more stable learning okay but to make this work the critic can't just be a standard Q function right a normal critic just looks at the state and the final action this one needs to guide a process exactly right it needs to be more sophisticated and that brings us to the specialized component, the noise level critic, which they write as Q5s at 2 ,5.

7:53Q5s at 2 ,5. Okay, what's the difference? Unlike a standard critic, 2 ,5, this one takes two extra inputs. It conditions on the noised action at a particular step in the process and the diffusion time step, which basically tells you how much noise is left. Right. So it knows not just the state, but also the current partially noisy action and how far along the denoising path we are. Why is that important? Why condition on noise? It seems a bit counterintuitive, doesn't it? But the key is that this noise level critic isn't just estimating the value of the final clean action. It's estimating the potential value even when the action is still noisy.

8:28Ah, okay. Like predicting the value of the finished product while it's still being made. Sort of, yeah. Imagine the action is a blurry photo. A standard critic can only judge the final sharp photo. This Q dollar can look at the blurry photo and, knowing how much blur there is from two, estimate the value of the final sharp photo it will eventually become. Okay, got it. It sees the potential. Exactly. And this enables something they call value transport. The way they train this critic forces it to be consistent across the whole denoising chain. The value estimate for a noisy action at time is trained to predict the value estimate for the slightly less noisy action at time, T10, all the way down to the final clean action at time zero.

9:10So value information flows backward through the process. Precisely. It propagates the final expected reward back through all the intermediate noisy steps. This has a really important smoothing effect. It stabilizes the gradients, makes the value landscape less bumpy, especially in noisy regions. And that stability means the critic's guidance signal, that gradient we talked about, is reliable. Very reliable. Even when the action is quite noisy early on, the critic can still confidently point towards the high-value regions, towards those good final actions. Which brings us back to that multimodal thing.

9:43The whole reason people looked at diffusion models was for exploring diverse solutions. Can this lightweight ACA, relying only on the critic, still do that? Amazingly, yes. And this is where the paper has some really compelling visuals. They tested it in a 2D banded environment specifically designed with four different equally good reward peaks for optimal ways to behave. Okay, a clear test for multimodality. And ACA nailed it. It achieved an aggregate sample proportion score of like 0.993. That means it found and consistently sampled from all four high-value modes, pretty much dividing its attention equally among them, about 0.25 proportion for each.

10:21Wow. And what about those complex diffusion baselines, the ones that were supposed to be good at this? They suffered from mode collapse, which is a common problem. Methods like DASER, despite being much heavier architecturally, basically just found one of the four modes and got stuck there. They scored 1.00 for one mode and 0.000 for the others. So ACA, without the explicit complex actor, actually did a better job at diverse exploration in this case. In this specific test, yes. It seems the critic-guided process, because it's stable and directly uses the value landscape, is inherently good at identifying all the peaks, not just getting stuck on the first one it finds.

10:59It retains the expressivity without the weight. Okay, that's impressive. Let's move from the illustrative bandit to the standard benchmarks. Mujoco continuous control tasks. How did ACA fare there? Empirically, it holds up very well. Looking at the learning curves in the paper, ACA generally shows faster performance gains. It often reaches high scores with fewer interactions compared to standard SAC, which is a really strong baseline, and also compared to the diffusion-based methods. Faster learning. Likely because of that eliminated policy lag. That's the most plausible explanation, yeah. Especially in the early stages of training, like on tasks like Hopper V4, you see ACA pulling ahead quicker.

11:39The direct guidance from the critic just seems to accelerate things. And let's hammer home the efficiency point again. The parameter savings. It's really significant. In Humanoid V4, which is notoriously complex, ACA uses only about 0.677 normalized parameters. ACC uses 1.00 by definition. That's nearly a third fewer parameters. Which translates directly to... Less memory, potentially faster training iterations on the critic side since there's no actor to train, easier deployment on hardware with constraints. It's a big deal if you're not running on massive GPU clusters. Right. Right. And there was another interesting result about robustness, particularly when dealing with suboptimal offline data.

12:19That's usually a killer for RL algorithm. Absolutely. This is a really interesting experiment. They tested ACA in an offline to online setting. So you start with a fixed data set of maybe not so great past experiences, and then you let the agent learn online from there. It's a tough scenario. Very tough. And they compared ACA against algorithms specifically designed for this, like CQL, conservative Q learning, and IQL, implicit Q learning. These often use pretty complex machinery to avoid exploiting bad data, sometimes using ensembles of like 10 Q networks. Okay, so heavy-duty offline methods and ACA.

12:55ACA just used its standard setup, pure online learning starting from that offline buffer, and only using a simple double Q setup just to critic networks, which is standard practice for stability in many online algorithms. And the result. Despite being architecturally much simpler, ACA matched or even outperformed CQL and IQL on tasks like half-T to V2 in this setting. Wow, so the inherent stability from the critic-guided denoising makes it robust even to poor starting data, without needing all the extra offline-specific machinery. It appears so. It suggests the core mechanism is just fundamentally quite stable and less prone to the pitfalls that usually require complex fixes in offline RL.

13:33Okay, so let's try and synthesize this. ACA seems to cleverly combine the expressive power people sought from diffusion models, that ability to handle multimodal actions, with the efficiency of drastically simplifying the architecture by removing the actor. And in doing so, it solves that fundamental policy lag problem of traditional actor-critic. That's a great summary. It's a synthesis of, yeah, expressive potential and architectural simplicity. Getting the best of both worlds, maybe. But it can't be all perfect, right? Yeah. What are the limitations or tradeoffs mentioned? Right. The paper is up front about this.

14:09The main tradeoff comes during action sampling. While the architecture is simpler and learning might be faster overall, the process of generating a single action involves that iterative denoising process. Taking multiple steps to go from noise to a clean action. Exactly. They found around T20 steps was optimal for performance versus efficiency. But each of those steps requires querying the critic network and doing a gradient population. So generating one action in ACA is computationally more expensive than just doing a single forward pass through an actor network like an SAC or PPO. Ah, okay. So faster learning overall, potentially, but slower decision making at inference time.

14:45That's the tradeoff. If you need extremely low latency, like millisecond responses for a robot, those 20 steps might be too slow compared to a single step policy. Makes sense. Any other caveats? The other main one is the guidance weight, that parameter W in the equation we mentioned. it balances how strongly the critic's gradient pushes the action versus sort of letting the denoising process just smooth things out. Right now, that W needs to be tuned manually for each environment or task. Hyperparameter tuning, the eternal challenge. Always. They found values like W30 or W50 worked well in their tests, but automating that tuning or finding a more adaptive approach is definitely future work.

15:25Okay. So powerful idea, impressive results, significant efficiency gains, but with a known trade-off in sampling speed and a key hyperparameter to tune. That seems fair. Which brings us to a final, maybe provocative thought for you listening. Given how well ACA seems to work, tightly aligning value estimation and action generation without needing that separate policy network, what does this imply? What broader assumptions about how complex our architectures need to be might this actor-critic without actor force us to reconsider? Are these dual network designs always necessary if one clever network can implicitly handle both roles.

16:00Something to think about.