


11 Jun 2025 · 18m · 12 chapters

The episode explains MAAS (Multi-Agent Architecture Search via Agentic SuperNet), a framework for automatically designing multi-agent LLM workflows that adapt per query instead of using one fixed “best” architecture.
No guests are identified in the transcript; it’s presented as a host/podcast discussion.
Early multi-agent systems (e.g., CAMEL, AutoGen, MetaGPT) required heavy manual prompt/role design. Automated approaches often optimize one static workflow, causing inefficiency (resource constraint) and poor cross-domain performance (cross-domain challenge). MAAS uses an “agentic supernet” plus a controller network to sample query-specific subnetworks, using an “early exit” operator to stop computation for easy tasks. It learns both architecture probabilities and operator behavior via performance/cost feedback and “textual gradient” (language-based improvement suggestions).
GAIA benchmark (file reading + web searching) where MAAS averages ~20.7% vs ~8% (AFlow) and ~6.7% (AD). Math example: slightly higher accuracy (~51.8% vs ~51.3%) with far lower cost (e.g., ~15% of AFLO training cost; ~25% inference cost).
Written by AI. May contain mistakes.
Discussion on the advantages and challenges of multi-agent systems in solving complex tasks.
“And like our sources mentioned, we have seen multi-agent systems boost capabilities quite a bit.”
Exploration of the resource constraint and cross-domain challenges faced by automated multi-agent systems.
“The research we're digging into highlights two key dilemmas, as they call them.”
Introduction of the Multi-Agent Architecture Search (MAAS) framework and its innovative approach.
“Either you're painstakingly building custom teams for every little scenario, or you automate finding one rigid, expensive team that's inefficient and not very versatile.”
Detailed explanation of agentic operators and how they contribute to optimizing workflows.
“The SuperNet itself is described as a cascaded multilayer workflow.”
How the controller network samples the optimal architecture based on query characteristics.
“And how does it actually get that efficiency for the simple tasks?”
Discussion on how MAAS learns from feedback to optimize both architecture and operators.
“First, updating the probability distribution.”
Review of MAAS's performance across benchmarks and its significant cost efficiency.
“That joint optimization allows the system to potentially discover completely new and highly effective ways for agents to behave and work together.”
Explore the performance differences and cost efficiencies of the new model.
“Math used only 15%, won 5 % of AFLO's training cost.”
Learn about the model's strong transferability and inductive capacity for new tools.
“It makes deploying these sophisticated multi-agent systems at scale seem much more, well, practical, feasible.”
Understand how the MLO framework could change collaborative AI systems.
“A strong ability to generalize and potentially adapt to new capabilities.”
Discuss the potential of making AI more accessible to a wider audience.
“The paper uses this phrase, democratizing access to intelligent automation.”
Speculate on the future challenges AI might tackle with these advancements.
“That's what solves those fundamental dilemmas of cost and flexibility.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Building these really sophisticated AI systems with large language models is just incredible, isn't it? It really is. I mean, think about what even a single powerful LLM can do now. Write, reason, translate. Even code pretty well. Yeah, the capabilities are amazing. But the real cutting edge, you know, where AI starts tackling genuinely complex multi-step problems, that seems to be when you get multiple LLMs, multiple agents working together. Exactly. Like assembling a whole team. Right. A team to solve something one expert couldn't really crack alone. Exactly. Imagine needing a system that has to, say, read a complex document, then maybe search the web for related info, analyze some data, and then draft a report.
0:43Okay. That takes coordination, right? Different schools coming together. And like our sources mentioned, we have seen multi-agent systems boost capabilities quite a bit. In things like Q &A, data analysis. Question answering, data analysis, code generation, yeah. Those areas have seen real benefits. But those early attempts building these multi-agent systems, it sounds like it was a bit like trying to handpick every single player for a championship team every single time. That's a good analogy. It required a ton of manual effort. Like what kind of effort? Well, painstakingly crafting prompts, defining really rigid roles for each agent, setting up exactly how they communicate.
1:21systems like the early camel autogen or meta GPT they often relied heavily on this kind of labor-intensive manual design as the paper we're looking at puts it right so naturally people tried to automate that yes the research community started automating parts of this searching for the best way to combine agents right but there was a catch most automated methods still tried to find one single ideal workflow one complex static multi-edging system optimized for maybe a specific type of task or a particular data set. So it's like finding the perfect blueprint for one incredibly specialized machine.
1:57Yeah. But then you're stuck using that one machine for everything, even tasks it's like wildly overqualified for. Precisely. And this leads to some major headaches. The research we're digging into highlights two key dilemmas, as they call them. Dilemmas. Okay. What's the first? First, there's the resource constraint dilemma. A single, super-optimized, complex system might be great at hard problems, sure, but it's resource-intensive, lots of LLM calls, maybe tool usage, high token counts. It costs a lot to run. The paper makes this point really vividly, actually. It says something like, PhD-level abstract algebra may indeed require complicated, token-heavy systems, while simple elementary-level arithmetic works well with a single zero-shot I.O.
2:44Ah, so if your only tool is that PhD-level system, you're just burning resources, energy, money on simple math problems. Exactly. You don't need the whole expert panel, like you said earlier, to figure out 2 plus 2. Okay, that makes total sense. What's the second dilemma then? That's the cross-domain challenge. Basically, a single static architecture, even a really powerful one, just isn't optimal across genuinely different types of tasks. So, good at one thing, maybe not so good at another. Kind of. The paper uses the GAIA benchmark as an example. It notes that there is no single system that is optimal for both file reading and web searching tasks.
3:23Those require different approaches. So if your automated method only finds one system configuration, you might have to completely re-optimize or build totally separate systems for different capabilities. Which isn't practical if you want, say, a general AI assistant that can handle whatever you throw at it. Exactly. It's not flexible. So you're kind of stuck. Either you're painstakingly building custom teams for every little scenario, or you automate finding one rigid, expensive team that's inefficient and not very versatile. Yep. That sounds like a fundamental bottleneck, right? It is. In scaling up intelligent, multi-agent AI.
3:58It really does. It absolutely is. And this is where the paper we're diving into introduces a genuinely different approach. It presents a framework called MAAS. M-A-A-S. It stands for Multi-Agent Architecture Search via Agentic SuperNet. And the core insight here is a fundamental shift. It moves away from trying to find one perfect monolithic system. Right. You mentioned that shift. So, Agentic SuperNet. Break that down for me. What does optimizing a SuperNet actually mean here? Okay. So, instead of optimizing a single blueprint for one building, think of MA as optimizing what the paper calls a probabilistic and continuous distribution of agentic architectures.
4:37Whoa, okay. Probabilistic distribution. Yeah, think of it less like finding the single best team composition and more like optimizing a flexible factory. A factory that contains all the potential players, all the positions, all the strategies. Okay, like a playbook. Sort of, yeah. A dynamic playbook and factory combined. And it can quickly assemble the right team composition and game plan for each specific problem that comes in for every single query. Ah, okay. So it's optimizing the capacity to generate diverse solutions adaptively rather than just finding one fixed solution. Exactly. That's the paradigm shift.
5:14The SuperNet itself is described as a cascaded multilayer workflow. It's built from these things called agentic operators. Agentic operators. Yeah. Yeah. These are basically the building blocks, the skills or strategies and an agent or maybe a group of agents might use. Like different tools in the toolbox. Exactly. And at each layer or potential step in this workflow, there are probabilities that determine which operators are likely to be useful or chosen. OK. What kind of agentic operators are we talking about? Are they like standard AI techniques? Some are becoming standard, yeah. The source lists several that My Ass uses.
5:50Coty, which is chain of thought for step-by-step thinking. Right. We hear about that a lot. Debate, where different agents might argue points to find a better solution. React, which involves reasoning and acting in an environment. Okay. Then there's self-consistency, self-refine, ensemble methods for combining results, testing, and a really important one, early exit. Early exit. We'll come back to that, I bet. So these are the tools in the factory's toolbox. That's a good way to put it. Okay. So the SuperNet represents this huge possibility space of different multi-agent workflows. Right. How does Moz actually use it to solve a specific query?
6:28Right. This is where the dynamism really comes in. During inference, so when a new query actually arrives, Moz uses something called a controller network. A controller. Yeah. And based on the features of that specific query, you know, how hard it looks, what domain it's in, maybe keywords, this controller network samples a specific multi-agent system. Samples. Like it picks and chooses. Exactly. It picks a customized subnetwork, a specific pathway from the supernet. So it looks at the query and thinks, okay, this problem needs this specific combination of operators arranged in this particular way.
7:03Pretty much. And that's what solves those dilemmas we talked about earlier, the cost and the flexibility. Precisely. Because the sampling is query dependent, MOS doesn't use the same heavy, rigid system every single time. Right. For simple queries, it can sample a very lightweight architecture, just a few steps, maybe simple operators. But for complex, maybe cross-domain queries, it samples a more intricate one that might involve multiple types of operators or go through more layers. And how does it actually get that efficiency for the simple tasks? You mentioned that early exit operator. Yes, that's key.
7:36The paper shows how MYAS uses this operator to dynamically decide the depth of the workflow needed. How many steps are actually required? How does that work? Well, looking at their case studies, for easy math problems, for instance, the controller network learns to give that early exit operator a really high probability early on, maybe at the second layer. Meaning the system quickly looks at the problem, realizes, hey, this is simple. Yeah. And then it samples the early exit instruction and just stops processing. It doesn't go through all the other potential complex steps. Yes, exactly. It avoids engaging the full, potentially complex machinery downstream.
8:13That saves significant computation, time, and cost. Smart. But for harder queries, the sampling probability automatically shifts. The controller recognizes a deeper process as needed, so it samples more layers and maybe more complex operators. So it adapts the depth. It adapts the depth, and it can even sample multiple operators to run within a single layer if the query seems to benefit from that. Like the paper shows an example combining IO for file access and a React loop for interaction, both happening at the same level for a tricky query. That's incredibly clever. It's like the factory doesn't just build the right tool, but it knows exactly how much tool is needed for the job at hand.
8:52And it stops building as soon as it's sufficient. Yeah. Okay, but how does this factory, this SuperNet, how does it know which combinations work best? How does it learn to build the right tools in the first place? Great question. It learns MacAxe optimizes the entire SuperNet. The whole thing. The whole thing. Both the probability distribution, so which architectures to sample, and the operators themselves, the building blocks. And it does this based on environmental feedback. Environmental feedback meaning, did it work? Basically, yeah. Did the sampled architecture perform well on the task? And crucially, how much did it cost to run?
9:28Ah, performance and cost. That feedback loop is absolutely critical for solving that resource dilemma, isn't it? Absolutely. It has to learn to be efficient, too. So how does that optimization actually happen? Is it like normal machine learning? It's a bit more complex. It happens in two main ways. First, updating the probability distribution. Okay. Learning which architectures are good. Right. Using techniques related to Monte Carlo sampling, the system learns which sampled architectures tended to be successful and cost effective for different types of queries. It then reinforces the likelihood of sampling those good, efficient combinations again in the future for similar queries.
10:06So the factory learns which assembly lines, which combinations of tools are best to favor for which types of orders coming in. That's a perfect way to think about it. And the second part is updating the operators, the actual building blocks themselves. Making the tools better. Well, see, these operators often involve things like natural language prompts or maybe interacting with external tools that are like black boxes. You can't easily update those with standard numerical gradients like in typical deep learning. Right. You can't backpropagate through a text prompt easily. Exactly. So this is where the paper introduces a really interesting concept, textual gradient.
10:42Textual gradient. That sounds intriguing, like AI is giving itself instructions in English. Essentially, yes. It's an AI-driven process, often using another agent, actually. This process provides feedback in textual format in natural language, suggesting improvements to the operators. Wow. Can you give an example? Sure. The source suggests the textual gradient might say something like, modify the prompt used by this operator to be more specific about the output format, or try adjusting the model temperature for this React agent, or even suggest structural changes, like consider adding another agent to the debate operator for this type of problem, or maybe incorporate few-shot examples into the self-refine process.
11:23That is wow. So the system isn't just learning which tools to combine, but it's also learning how to actually make the individual tools smarter and more effective over time using language-based feedback. Exactly. The building blocks themselves evolve alongside the strategy for combining them. That seems incredibly powerful. That joint optimization allows the system to potentially discover completely new and highly effective ways for agents to behave and work together. It does. It opens up a lot of possibilities. Okay, this theoretically sounds brilliant. I mean, flexible, efficient learning. It ticks all the boxes.
11:59But, you know, the proof is in the pudding? Does it actually work in practice? What do the experiments show? Right. The theory is nice, but the results, they're quite compelling, actually. The researchers evaluated Maas across six different benchmarks. Covering what kinds of tasks? Things like math reasoning, code generation, and complex tool use situations where multi-agent systems are often needed. And the headline outcome. Did it perform well? Extremely well. The paper states Maas has consistently achieved optimal performance across all task domains they tested. Optimal. Wow. Yeah. It surpassed the existing methods they compared against by a range of, let's see, 0.54 % up to 16.89%.
12:38That's a pretty significant jump, especially at the higher end. It really is. And look at the GAIA benchmark. That's the one testing complex tool use needing that cross-domain flexibility we talked about. File reading and web searching, right? Exactly. On GAIA, Moss achieved an average score of about 20.7%. Compare that to other automated methods they tested, like AFlow at 8 % or AD at around 6.7%. Okay, that's a huge difference, 20 % versus 8%. Massive. And the paper directly attributes this stark difference to MAS's adaptive nature, its ability to select different architectures for different tool use scenarios within the same benchmark.
13:14It wasn't stuck with one approach. Okay, so definitely better performance, especially on complex, mixed tasks. That's important. But you really emphasize the resource constraint, the cost issue. How did it do on efficiency? Ah, yes. This is where the results become truly, truly striking. The paper reports that Moss requires significantly lower inference costs. Get this. Only 6 to 45 percent of the inference costs of existing handcrafted or automated multi-agent systems. Wait, hang on. Did you say 6 percent? As low as 6 percent of the cost in some cases? That's what the paper reports. Yeah, 6 percent.
13:48That's not just incrementally better. That's a completely different ballgame. It really is. Let's make it concrete with the math benchmark results they shared. Okay. On math, math actually achieved a slightly higher accuracy than a comparable automated method called AFLO. So 51.8 % versus 51.3 % roughly. That's a slightly better performance. Slightly better performance, yes. But the cost difference, it's enormous. Math used only 15%, won 5 % of AFLO's training cost. Wow, training cost too. Training cost, yeah. About$3.40 versus$22.50. And even more critically for deployment, just 25 % of AFLA's inference cost per query.
14:26If so, a quarter of the cost will run it. A quarter of the cost. About$0.42 versus$1.66 per query on average. And it also used fewer tokens and had shorter wall clock times. That perfectly demonstrates the power of only using the computation that's actually necessary for each query, thanks to that adaptive sampling and early exit. Exactly. It shows it's not just slightly better, it's much cheaper and faster for a comparable or even better performance. That kind of cost saving is genuinely transformative. It makes deploying these sophisticated multi-agent systems at scale seem much more, well, practical, feasible.
15:03Absolutely. It could unlock a lot of applications. And the results also showed other promising signs. Like what? Strong transferability. The optimized supernet, the factory, worked well even when they swapped out the underlying large language model using things like GPT-4 and Mini, QN, LAMA. So it's not tied to one specific LLM. It doesn't seem to be, no. And it transferred effectively to new data sets it hadn't seen during the main optimization. Plus they found something they called a surprising inductive capacity. Inductive capacity, meaning? Meaning Massas could even make effective use of agentic operators, those building blocks that were unseen during its training.
15:41Whoa. So you could potentially give it a new tool later on and it might figure out how to integrate it effectively. That's what the results suggest. Yeah. A strong ability to generalize and potentially adapt to new capabilities. OK, let's pull this all together. Let's bring this back to you, the listener. Why is this particular piece of research, this MLO framework, why is it a big deal for what you might see or use in the future? Well, because the future of really powerful AI assistants, the kind that can help with complex tasks, isn't just going to be a single giant AI brain, most likely. It's often going to involve intelligent collaboration between different specialized components or agents.
16:22But traditionally, building those collaborative systems hit these practical limits. They were too rigid, too expensive to build and run. Okay, that makes sense. And MA offers a potential way past that bottleneck. It flips the problem on its head. Right. Instead of painstakingly designing one perfect static system. It optimizes the ability, the factory, to dynamically assemble the right system, the best configuration for each specific task right when it's needed, on the fly. Which means, what for the user? Which means you could start to see AI systems that are far more versatile, capable of tackling really complex problems that need diverse skills like the code generation, the multi-step math, the tool use examples, and also dramatically more efficient and affordable to actually run.
17:06The paper uses this phrase, democratizing access to intelligent automation. Democratizing access. Yeah. Meaning making this sophisticated AI power potentially accessible to a much wider range of users, businesses, applications than before. That's the implication, yes, because it tackles that cost barrier. It really is that shift from a single, highly specialized, expensive machine to a self-optimizing factory that builds the exact right tool needed for this specific moment. Carefully balancing capability with cost. That ability to adapt the architecture itself based on the query, that seems to be the core innovation here.
17:43That's what solves those fundamental dilemmas of cost and flexibility. So thinking about that, if AI systems can really learn to build the best version of themselves for each specific problem they encounter, doing it dynamically, efficiently, and potentially a fraction of the cost of previous methods, what incredibly complex challenges might they be able to tackle next? And maybe more personally, how might that fundamental shift change the tools you rely on every single day?
This paper introduces MaAS, a novel framework for automating the design of multi-agent systems built on Large Language Models (LLMs). Instead of seeking a single best system, MaAS optimizes an agentic supernet, a probabilistic distribution of possible architectures. This allows MaAS to dynamically sample query-dependent multi-agent systems, tailoring solutions and resource allocation based on the specific input. Experimental results demonstrate that MaAS achieves higher performance across various benchmarks compared to existing methods while being more resource-efficient in terms of training and inference costs. Furthermore, MaAS exhibits strong transferability across different datasets and LLMs and possesses inductive capabilities to handle new agentic operators.