Agentic Economic Modeling

0:00Welcome to the deep dive. Today we're digging into something called agentic economic modeling or AEM. It sounds technical, but it might just shake up how huge companies make those really big billion-dollar decisions. So imagine you're running a massive online business. You constantly face choices, right? Like, should we tweak our return policy? Or maybe offer a faster shipping option? But will that just steal sales from our standard shipping? Or will it bring in new business? Answering that means understanding cause and effect. You need to know the what-ifs, the counterfactuals. What would have happened if we didn't make that change?

0:36And usually getting reliable answers means, well, a lot of hassle. We're talking big, expensive, slow methods. Think massive randomized control trials RCTs where you change things for millions of people for weeks. Or maybe exhaustive market surveys. Takes ages. Costs a fortune. Exactly. And that's why, you know, when large language models, LLMs, came onto the scene, everyone got excited. It looked like a potential shortcut. A shortcut how? Well, the promise was, maybe we can skip collecting all that expensive human data. The idea was kind of simple, really. Treat the LLM like an economic agent, like a homo economicus.

1:09You give it a persona, tell it the situation, and just ask it what it would choose. So you could simulate, like, a million customers instantly. Theoretically, yes. Yeah. And eliminate all that sampling uncertainty that plagues smaller surveys. It sounded almost too good to be true. Almost. Okay, I sense a butt coming. So why didn't RCTs just disappear overnight? Because reality bites, I guess. Yeah. As soon as people tried using those raw LLM outputs for actual economic predictions, they hit a serious wall. A wall. But these models know so much about language and, you know, human preferences from all the text they've ingested.

1:44How could they fail at pretending to be a shopper? Well, they have the knowledge, sure, but their choices are systematically different from real human choices. They show consistent bias. Bias like how? Often it's like a monotone preference distortion. Maybe the LLM agent always picks the cheapest option or always the fastest, no matter the persona you try to give it. And critically, they just don't have the same variety, the same heterogeneity you see in a real population. Ah, so everyone kind of acts the same in the simulation. Pretty much. You get this huge data set. But it's, well, it's flawed.

2:14Using those raw LLM choices, fundamentally unreliable for making real economic predictions. And that's the problem. Agentic Economic Modeling AEM was designed to tackle. It's a framework, a rigorous three-stage process specifically built to correct that systematic bias. Okay, so the mission is to take the LLM's vast knowledge, which is there but kind of skewed, and anchor it back to reality. Precisely, but using only a small amount of real human data to do the anchoring. The goal is to turn that unreliable generator into a trustworthy economic simulator. Right, let's unpack that. You said three stages, generation, correction, and inference.

2:52Let's start with stage one. Generation. Sounds straightforward enough. It is. Mostly. You use the LLM to crank out loads of synthetic decision data. But not just one type of decision. You create diverse personas simulating different kinds of customers and see how they react under your specific experiment conditions. So you're not just getting one prediction, but like a whole panel of synthetic viewpoints. Exactly. Capturing different potential decision rules. Which leads us nicely into stage two correction. And this is really the heart of AEM, the core innovation. This is the magic step that fixes the LLM's flaws.

3:30Essentially, yes. It transforms those knowledge-rich but biased predictions into precise estimates that actually align with human behavior. And you mentioned earlier this connects to some classic economic ideas. Random coefficient discrete choice. Sounds heavy. It does, but the core idea is intuitive. The researchers realized, look, trying to build one perfect synthetic agent is probably a losing game. It's too hard. Instead, they borrowed this idea from mixture models. Think about any group of real customers. It's not one monolithic block. It's a mix. Sure. You've got bargain hunters, quality seekers, people who want speed, all sorts.

4:08Exactly. A mixture. Some are price sensitive, some value convenient, some are loyal to a brand. And the community's overall choice pattern isn't driven by one rule, but by a weighted combination of all these different decision rules. Ah, okay. So you're saying. So with AEM, we pre-generate a set of diverse synthetic LLM personas. Some might be flawed, some biased in different ways. Then, and this is key, we use a small amount of real human data. Just a little bit. Just a little bit. We use that real data to figure out the optimal weights for each synthetic persona, like finding the right recipe.

4:42How much of the price-sensitive agent do we need? How much of the speed-focused one? To perfectly match the choices we see in the real limited human data. Huh. So instead of fixing the broken robot, you built a team of slightly quirky robots and just figured out how to blend their outputs correctly. Is that complexity really better than just getting more human data? It is. Because getting lots of human data is the expensive, slow part we want to avoid. We leverage the fact that generating synthetic data is cheap and fast. We make tons of it. Then we only need that small human sample to learn the right blend, the right weights.

5:15The cleverness is in that learning step, not in trying to perfect the LLM itself. Got it. That makes the value proposition much clearer. Cheap simulation, small calibration. Okay. So once you have those weights and the corrected data, what's stage three inference? Right. Stage three, inference. Once you've applied the correction and you have this refined synthetic data set, let's call it ANCHI, you basically treat it just like it was a complete real world human data set you'd spent months collecting. So you can just run your standard analysis on it? Yep. Standard econometric models, causal inference methods, whatever you normally use.

5:47Calculate demand elasticities, estimate treatment effects, the works. But based on data that was mostly generated synthetically and then corrected. Okay. Theory sounds promising, but does it actually work in practice? You mentioned validation cases. Absolutely. They put AEM through its paces in two pretty demanding real-world scenarios. Let's hear about the first one, conjoint analysis. Yeah. The first was a huge conjoint study. That's the classic market research technique, you know, where you show people different product versions, different features, different prices, and ask them which they prefer.

6:21Right. It helps you figure out what features people actually value, but notoriously slow and pricey. you said. Exactly. Costs thousands, takes weeks just for one product sometimes. So in this experiment, they had millions of real human observations already. The goal was, could AEM drastically reduce the need for that human data? How much did they try to reduce it by? They used only 10 % of the original human data, just 10%. That was used to fit the bias correction model, to learn those crucial weights we talked about. Then they used the corrected LLM choices to simulate the other 90 % of the responses.

6:54And the results, did it work? Well, first they tried the naive approach, just using the raw LLM outputs without correction, maybe weighting them simply, and get this, it was worse than doing nothing. Using the raw uncorrected LLM choices actually increased the error compared to just using the small 10 % human sample alone. Wow. Okay, so that's a huge warning sign. Naive AI augmentation can actually hurt your decision making. Absolutely critical point. The correction stage isn't optional. It's essential. So what happened when they did use the full AEM with the correction stage? Then the picture totally flipped.

7:30The bias corrected estimates were significantly more accurate. The best corrected estimators cut the bias by over 16 % compared to just using that initial 10 % of human data. 16 % reduction in error. That's substantial. It really is. It means your forecasts about how customers will react are much, much more reliable. It proved AEM could effectively amplify a small amount of human data into high-fidelity insights. Okay, impressive for predicting preferences. But what about real-world behavior? You mentioned a second case, a field experiment, an RCT. Yes, and this one I think is even more compelling.

8:03It's about scaling AEM in a live randomized control trial setting. This was a regional experiment dealing with delivery options for an e-commerce platform. And these kinds of RCPs are usually massive of undertakings. Yeah, huge. You often have to roll out the change to, say, 10 percent of your regions, maybe more, and let it run for eight weeks, maybe longer, just to get statistically reliable results. It's costly, operationally complex, and impacts real customers. So what was the specific goal in this delivery experiment? The specific outcome they measured was the change in the share of customers choosing same-day delivery.

8:39The ground truth, what they found when they ran the full expensive human experiment across the whole country for weeks, was that the treatment caused a decrease of 60 basis points. So a 0.6 % drop in same-day share. Okay, so minus 60 basis points is the number to beat. How did AEM try to get there more efficiently? They tested it two ways. First, scenario A, region-wise extrapolation. Could they reduce the scale, the geographical footprint of the experiment? How did they set that up? Super clever. They ran the human part of the experiment in only 10 % of the region specifically, 10 % of the IP3s nationwide.

9:11wide. That small sample was the in-domain data used to calibrate the AEM model. Okay, just 10 % for calibration. Then, the model had to predict the treatment effect for the other 90 % of the country, the out-of-domain regions, where they had no human experimental data. That sounds like a massive leap. Predicting for 90 % of the country based on just 10 %? How did it do? Shockingly well. The AEM mixture model estimated the effect in those untested, out of domain regions was minus 65 basis points with a confidence interval of plus or minus 10 basis points. Wait, minus 65? The real answer was minus 60.

9:47That's incredibly close. Right. Only five basis points off the national truth. That's five one hundredths of a percent difference predicted for areas they never actually tested in the real world. The generalization was fantastic. Just think of the savings. You could potentially run a much smaller test and still get the national picture. Okay, that's reducing scale. What about reducing the time? Scenario B. Ah, yes. Scenario B. Timewise extrapolation. This is where I think you really see the power. The question was, could AEM let you stop an RCT much, much earlier than the usual eight plus weeks?

10:21How early did they try to go? They pushed it to the extreme. They tried calibrating the AEM model using just one single day of human experimental data. One day. That's nothing, right? For a major business decision based on an RCT, one day of data is usually just statistical noise. Totally useless on its own. And the results proved it. Looking only at that single day of human data, the estimated treatment effect was weak minus 17 basis points. And crucially, the confidence interval was enormous. Negative 43 basis points all the way to plus 9 basis points. So statistically insignificant. Could be bad, could be good, could be nothing.

10:56You can't make a call based on that. Exactly. P-value is high, around 0.2. You wouldn't act on that. It's noise. But then they added the AEM layer, using that same single day of human data just for calibration. Yes. They combined the insights from the LLM simulations, corrected using that one day of human calibration data, and the picture changed completely. The AEM augmented estimate came out at negative 24 basis points. Okay, still a bit off the final 60, but look at the confidence interval. It shrunk dramatically to negative 26 basis points, negative 22 basis points. Oh, from negative 43 plus 9 down to negative 26, negative 22.

11:29That's incredibly precise. Incredibly precise. And instantly, highly statistically significant. The p-value plummeted to less than 1 in 100 ,000. That is the aha moment right there. One day of human data. Useless noise. One day of human data plus corrected LLM simulations. A statistically rock-solid precise estimate. You nailed it. It takes you from complete uncertainty to actionable insight, potentially cutting weeks of expensive testing down to, well, maybe just a day or two for calibration. AEM delivered the statistical power and robustness that the sparse human data simply couldn't on its own.

12:04Okay, stepping back, what's the big picture here? What does this AEM framework really mean? Well, it successfully bridges that gap, doesn't it? Between the low-cost, massive scale of synthetic LLM data and the high-fidelity results you need for serious economic analysis results that used to require those big, expensive human experiments. So it's about efficiency, fundamentally. Huge efficiency gains. You can potentially drastically cut the scale, the duration, and therefore the cost of experiments without sacrificing the statistical rigor, the grounding and randomization that makes methods like RCTs so valuable.

12:38It sounds like a foundational method almost, like a new way to generate counterfactuals. I think that's fair. It establishes a principled way to use LLMs for counterfactual generation. Anytime you have a causal question where you need to know what would have happened if. But you can't easily run the full experiment everywhere or FAC. AEM offers a path. It's like a cheat code for filling in missing data in causal inference problems. A cheat code for cause and effect. I like that. It's a huge potential, clearly. But are there still challenges? Open questions. Oh, absolutely. While AEM showed remarkable accuracy and efficiency gains, especially in cutting scale and initial time, the research did flag one area where more work is needed, particularly in that time-wise prediction scenario.

13:21What was that? Well, think about that RCT running over weeks. The treatment effect often isn't static. It might change or stabilize over time. The current AEM approach, as presented, doesn't really have a built-in way to model those temporal dynamics explicitly. Ah, so it predicted the early effect well, but maybe not how it would evolve over, say, week two, week three, week four. Exactly. If the effect stabilizes, why does it stabilize? Is it because customers are adapting, learning, forming new habits? Or is it something external like new types of users coming in or maybe seasonality kicking in?

13:58The current framework doesn't inherently distinguish those drivers of temporal change. Right. So that leaves us with a final thought maybe for you, the listener, to chew on. Yeah, the question becomes, how could you adapt something like AEM to make accurate predictions, not just across different regions or customer segments, but accurately over time, especially when you don't know beforehand what those underlying temporal trends or adaptation patterns might be? Modeling that dynamic evolution robustly. That seems like the next frontier for this kind of powerful agentic modeling.