Benchmarks, borders and the true cost of AI regulation

0:04James Wilson: Hey everyone, I'm James Wilson and welcome to this solo Risky Business Features podcast episode. What an incredible moment it is right now where we simultaneously have some of the most capable, interesting and powerful open weight models being released by ZAI with GLM 5.2, Moonshots Kimi K3, we've got Quen 3.8 coming out on the horizon. But whilst we've got these most amazing models in our hands now, we've also simultaneously got the US government and the Chinese government signaling that they intend to take some sort of regulatory steps to limit access to or even prevent Chinese labs from being able to create these open-weight models.

0:47James Wilson: In this podcast episode, I want to look beyond just the rhetoric that's coming out of the US and Chinese governments and instead game out the sort of real world scenarios of what happens if the US makes it commercially non-viable to use open weight models, prohibits the use, or even takes steps to make it difficult, well maybe impossible, I'm not sure they really can make it impossible, but somehow significantly impede the ability for a Chinese frontier lab to create these open weight models. Because the reality is that although it may look like on paper there's comparable drop-in replacements available from Anthropic or from OpenAI or even from Meta, there is a significant cost and a number of challenges involved in actually swapping out an open weight model for a closed weight model.

1:39James Wilson: And that's the purpose of this podcast. We're really to get down to the depths of what is the real cost and who pays the cost if there are steps taken to limit access to or to try and impede the creation of these open-weight large language models. First let's examine what's actually been said in terms of what these restrictions might look like. Right now at time of recording on the 28th of July there is no concrete proposal that has been put forward, but rather what we've got is we've had messages on Twitter slash X from Michael Kratzios, the director of the White House's Office of Science and Technology, where he claimed that Moonshot AI had developed a sophisticated internal platform to conduct large-scale distillation against US models, and that large-scale covert industrial distillation aimed at stealing proprietary US technology and undermining American research is unacceptable.

2:36James Wilson: Now, that did not contain a specific threat as to restriction, but then that was backed up by Treasury Secretary Scott Besant, who also weighed in on X, saying that sanctions and entity list designations will be on the table. So again, no specifics, but a flavor of the fact that there is discussions being held, there is a view that this is unacceptable, and that the measures called out could be sanctions and entity list designations. Now, what we know about the Chinese government's feelings here is not quite as clear, but there was reporting from Reuters that Alibaba, ByteDance and ZAI had attended meetings with authorities and those meetings centered around the suggestion that potentially leaks or thefts of AI could become punishable under national security law and that also that there could be restrictions on who can actually fund the Chinese domestic AI startups.

3:30James Wilson: But again, this is just sort of sources reporting at the moment there's nothing official that has come out. But it definitely feels like there is a desire on the US side to do some sort of, you know, regulate these open-weight models out of existence. Now, whether that's achieved by preventing the creation from them, prohibiting the use of them, that's not clear. And I guess that's an important point to mention there is that regardless of what form these restrictions might take, They are sitting upon existing measures that limit access to chips, that prevent certain exports to certain entities on that entity's list.

4:08James Wilson: And interestingly enough, ZAI, the makers of GLM 5.2, they're in fact already on that entity list. The other thing that has happened just in the last few days is that Senators, Banks and Schiff have introduced a bill called the Collaboration on Adversarial Threats and Security Risks Act, which in their words creates a narrow tailored antitrust safe harbor allowing AI companies, American AI companies, to share threat information and coordinate defensive measures in good faith to address defined national security risks. while preserving existing antitrust protections against price fixing, monopolization, and other anti-competitive conduct.

4:49James Wilson: Interestingly, the press release talks about this is to ensure that America must win the AI race. And while it's not clear what even winning that AI race looks like, throughout this podcast we'll look into the real cost of regulating access to open-weight models and how I don't think it actually serves that goal of America wanting to win the AI race at all. But that said, in the spirit of a balanced perspective, let me try to steel man why policymakers might think that, first of all, preventing access to these open-weight models or limiting the ability for them to be created would be something that actually doesn't force a cost upon the consumers of AI models in the US.

5:40James Wilson: It is conceivable that someone could look at the current offerings from Anthropic and OpenAI and see that there are both pricing tiers and capabilities that look like a pretty good match one for one for the various different varieties of open weight models. right we've got open weight models that are quite small 7 billion 9 billion parameters designed to run locally on small hardware we've got medium size let's say open weight models 27 billion maybe up to 122 billion that's kind of the limit of being able to run on let's say hardware with 128 gig of vram or unified ram and then you've got the much larger models right the 500 700 hundred, even the trillion parameter model, which really are only accessible if you've got access to clusters of very capable GPUs.

6:30James Wilson: But to steel man this, I can see the point of view where someone looks at even the high level capabilities of these open weight model offerings and could map them to the pricing tiers and capability tiers that are now offered by Anthropic and OpenAI. Anthropic has offered capability-based pricing for a while, right? The notion of Haiku, Sonnet, and Opus for different levels of capability and complexity of task have existed for a while. And of course, they are offered at different pricing tiers, and they have different capabilities in terms of context length. OpenAI has just entered that fray recently with their GPT-5.6 models.

7:09James Wilson: we've now got Sol, Luna, Terra, which are essentially the same sort of concept. They're different pricing tiers, different capabilities, and one would assume, although they don't really publish the details, these are analogous to those same sorts of different sizes of models that therefore have different requirements for inference. Some can be run cheaper, and some have more restrictions on their, I guess, capability around things like, again, context window, length, token speed, tool calling, etc. So on one hand, I do get the idea that it can look like there are drop-in replacements for open weight models.

7:46And I guess I'm suspending disbelief here a little bit.

7:50James Wilson: I'm saying, yeah, I can see that people could make the argument that, look, these models, they're all same, same. Look at the benchmark scores. They're like for like. Look at the token pricing and performance. They're like for like. You just drop them in and no problem. but there is a problem there is a hidden policy assumption here this idea that you can just ban hammer these open weight models it only works if the replacement model can be trusted with the data in the same way permits the same sort of cyber security tasks to be done supports the integrations and api endpoints that are required remains durably available and also has a model with the same behavioral characteristics that won't require a whole bunch of rework to allow this replacement model to be as effective.

8:37James Wilson: That's a lot more to consider than just token price and benchmark scores. And a lot of these requirements can't even be satisfied today. To navigate this, we need to fully consider what an open-weight model brings to market. Of course, terminology here, open-weight model refers to a set of files that are the actual floating point values that are at the heart of a large language model. It is the large language model. I've used this analogy before, though. It's, if you use the car analogy, the large language model, these open weights, are the engine block. They're not the whole car. When you're using an open weight model, there is still a lot of other machinery and tooling around it.

9:19James Wilson: There's a separate inference engine. There's an entire ecosystem of software that exists to get you from having possession of an open-weight model with hardware that can accommodate it to actually being able to serve real inference requests. The key thing is, of course, the open-weight model can be run locally. It can be run inside an air gap or it can be run in a chosen cloud environment. And this is what's important about them is that the open-weight model is not a cheap API or a cheaper way to do inference, it's a different architecture. The fact that these weights are downloadable and you have the choice of where to deploy them opens up all manner of flexibility that is simply not possible with a closed weight model.

10:06James Wilson: Let's look at the choice of cloud provider, for example. This could matter to a cybersecurity operator because perhaps there needs to be a certain degree of proximity between the rest of a particular solution and the large language model that it depends upon. You might be reliant on a certain trusted partner relationship with a cloud provider that you just can't get with a closed-weight model provider. Even an exotic thing like the feature set of the cloud provider that is hosting your open-weight model can matter. For example, there are cloud providers that will host open-weight models for you that offer private inference, that is, provably private access to these open-weight models, and that's just not possible with a closed-weight provider.

10:46James Wilson: With an open-weight model, the operator controls the data boundaries. They also control that software serving stack and the supply chain behind it. They control when the model and the serving stack is updated. But they also have the ability to further refine and enhance the open-weight model through fine-tuning and various other techniques that are sometimes referred to as post-training that allows an organization or an individual to take an open-weight model and to essentially impart more knowledge into it to make it even better at a specific task. Again, you just can't do that with a closed-weight model.

11:24And yes, on the flip side, the control that an open-weight model gives you does also bring the complexity of hardware costs, maintenance, supply chain concerns, governance costs. But this is where you've got to look at the trade-off. It may well be, and I can see many likely scenarios, where that cost is actually the price of making a cybersecurity workflow even possible, especially when you consider things like the data boundary. It's still just not palatable or possible or perhaps even too risky for a closed-weight model provider to see the conversations and the interactions that you're having with these large language models without potentially exposing proprietary IPs such as source code or, you know, exploit development that's underway, vulnerability research that's underway.

12:14You know, it's kind of funny. Some of the research I've been doing, I start off with Claude, Claude gives up and then I switch to GLM and then sometimes I switch back to Claude to get it to validate GLM's research and Claude freaks out at me. It's like, why did you expose this to GLM? This is now a risk. It has been in the hands of another lab and I'm like, buddy, that's my risk to manage. Don't you worry about it. But for me, that's okay. And in fact, I don't really have another choice. But you can imagine for organizations and enterprise, this is a very real concern. If the choice is an open weight model hosted in your way with your data boundary and security posture baked into it, versus a closed weight hosted model that you can't control who's seeing that, who's logging it, who's using the data for further training or even other product creation, well, there's no drop-in solution there.

13:08It's just, it's not viable in that case to switch from an open-weight model to a closed-weight model.

13:13James Wilson: So what does an open-weight model bring to market? It brings to market control over the deployment. That's the key thing here. It's a deployment property. It doesn't mean free, safe, unrestricted, or equally capable access to inference. The key point is you control the deployment of these models. So let's bring this to life with two examples. One I've covered in a previous Risky Business Features episode with Carsten Knoll, where we looked at his technique of using a locally hosted open-weight model to do code reviews, but having that smaller, lesser capable model being orchestrated by larger and more capable closed-weight frontier models.

13:55This is a great illustration of the value of an open-weight model. Because in this case, as Carsten says, code reviews remain one of the

14:04James Wilson: most high-value tasks you can do in terms of trying to secure a product or audit a product. But it's also something that gets taken off the table pretty quickly because people are too concerned about that IP going to a third party. A local open-weight model, if you can construct a code review apparatus such that it is that local model that is the only part of the system that has access to that source code, and if you're even then running that locally hosted model within the customer's infrastructure, then you've very cleanly mitigated that concern. This is an example of something that just hasn't been possible in general until there was the advent of a capable, locally hostable, open-weight model that could do code reviews to a point where it was good enough for a larger and more capable frontier model to then orchestrate and adjudicate the results.

14:57I do this exact same setup at home at the moment with some of the vulnerability research work I'm doing. I will use a DGX Spark, one of the NVIDIA GP10 devices, 128 gig of RAM. It hosts a QEN 3.6 27 billion parameter model, or I sometimes run a 122 billion parameter with a lower quantization. But it's the thing that I throw anything at really. I'm comfortable with it seeing anything that I want to look at because I know that I have, I guess, the stage gate to determine once it's given me its findings and its outputs, which of those do I want to be adjudicated

15:31James Wilson: directly by a frontier model and risk it exposing the research that I'm doing. Or I can sometimes say now take that finding and now create a more abstract description of it that I will then send that to the frontier model if it seems particularly sensitive. Again, optionality that is only made possible by a locally hosted open-weight model. Along the same lines, but leaving aside the data privacy concerns, it's also a much better division of labor and a very powerful cost control mechanism. So I am not in the position where I can afford very large bills from API usage. I rely on subscription pricing, and I do happily subscribe to many AI providers, and I try as best as I can to maximize my use of my five-hour windows and my weekly windows.

16:21James Wilson: Not to the point where it's becoming a source of anxiety, but, you know, it's always on my mind. Am I doing enough in this five-hour window right now to maximize my use? But that's the nature of subscription pricing. On the flip side, though, I love having a locally hosted open-weight model because it's almost the exact opposite to the subscription pricing. Whereas I have to think carefully about really high-value tasks to maximize the use of that five-hour window and that weekly window. With my local model, I just need to consider more what is like a long funnel of tasks that I can get it constantly churning away on.

16:57And even if it's going to result in a bunch of false positives or a bunch of things that are not going to be real findings anyway, it doesn't matter. It's worthwhile to have something locally constantly churning away, constantly generating hypotheses, constantly thinking of potential leads. because that's always-on near-zero-cost inference that I have available to me at home. And then I pair that with the frontier models to then do the adjudication, do the orchestration, which is a much better use of the constrained five-hour and one-week usage quotas that I have with those.

17:33James Wilson: I chose these two examples because they demonstrate what an open-weight model has made possible and therefore illustrates the downside of an open-weight model not being available for two very different entities. In the code review example, that is a large enterprise that was concerned about its IP going to a third party that would be disadvantaged if it could not leverage an open-weight model to do its code reviews. In the latter example, that's me as an individual that would be less capable as a software engineer or as a cybersecurity professional because of a lack of access to these open-weight models.

18:11So my point here is the costs and the disadvantages if there is restrictions on open-weight models apply to the big enterprises as well as the single individual operators and everywhere in between.

18:26James Wilson: Okay, so leaving aside the data boundary issues that we just went through, what else is it that means that an approved, closed-weight API for inference is not a drop-in replacement for an open-weight model either running locally or hosted somewhere in someone's cloud. I mentioned at the start of this episode that the idea that the US government could essentially regulate open-weight models out of existence by whatever means and not have that have a huge cost and detrimental impact on consumers of AI was false. And that's because it's based on a policy assumption that just doesn't stack up. A closed-wait API for inference only works as a drop-in replacement for an open-wait model if the following criteria are met.

19:14If it can be trusted with the data in the same way. If it permits cybersecurity work to even be done at all. If it supports the same sorts of integrations, and there's more than just API here. If it remains durably available, and there's a bunch of stuff to unpack there. But perhaps one of the biggest points here is that it's only a drop-in replacement if the model has very similar behavioral characteristics to the point where both tooling and existing data sets and harnesses don't need significant rewrite or even regathering of expensive evaluation runs in order to continue to produce the same quality of results.

19:53James Wilson: If we woke up tomorrow and we couldn't access nor use an open-weight model, this would be a massive problem, particularly for defenders in the cybersecurity space. And the reason for that is that you would be forced to use API inference through either Anthropic, OpenAI, maybe Gemini. I focus mostly on Anthropic and OpenAI because that's what I'm most familiar with. But for the base level individual who has not gone through their opaque approvals process for accessing models for cybersecurity work, you would no longer have access to a top tier frontier model to do cybersecurity work. In the current extremely guardrailed environments, neither GPT 5.6 nor Opus 5, certainly not Fable 5, they will not participate in anything that looks or smells or feels or remotely resembles cybersecurity work.

20:45And, you know, the guardrails are a bit different in different cases. Here's some of the things I've noticed. Fable 5 remains utterly useless. The moment you mention anything with cybersecurity or any sort of keyword around that, it just says no, kicks you down to Opus. Opus 5 will go a little bit further and it'll sometimes get you through the sort of research piece, but certainly once you start to really push it to chain things together to exploits, to proof of concepts, it gets pretty freaked out and will stop. GPT 5.6 is somewhere in the middle there. I have not been able to get it to successfully be my orchestrator for vulnerability research, But interestingly, it was happy to create my vulnerability research harness for me, despite the fact that it was clearly labeled and clearly signposted as being intentionally used to find vulnerabilities, to create exploits, to create proof of concepts that we could then use as a good guy to file those as bug reports.

21:40So there's an interesting spectrum there, but the amount of impediment that someone would face if they were only left to choose between OpenAI and Anthropic for cybersecurity work is immense if an open-weight model was not available right now. Yes, as I mentioned, you can go through their approval process to get access to a less guard-railed version of Opus. I have that at the moment. It's very capable. You can do the same for OpenAI. I've tried that. I don't get access. I'm not sure why. But the fact that these approvals processes are opaque and it's not clear exactly what bar you've got to clear or even what sort of timeline you should expect for approval, that's a pretty bad position to find ourselves in if tomorrow there were no open wait models.

22:27One of the other things that makes a closed wait API for inference not a drop-in replacement for open waits is the supported integrations. And there's two levels here. One is around the actual API surface itself. Open weight models, they tend to mimic either the Anthropic or the OpenAI style of transcript interaction, actual HTTP API interaction. But those things are, the two protocols there are quite different despite sharing a lot of similarities. is. If your current open-weight model is using the OpenAI style API and tomorrow your only option was to go and use Anthropic, then you're going to have to then rewrite a bunch of your harness and agent code to work with that new API.

23:11There's also the challenge around integration such as tool calling. You might have a set of tools that have been very fine-tuned for the patterns that you've seen in how an open-weight model tends to like to call those tools. Different models have different personalities, whether you like the anthropomorphizing there or not. They're just different beasts. And they behave differently in terms of you can give them whole suites of tools and some cope well when there's a large suite of tools, some don't cope well, you've got to really narrow it down. Some like specific ways of defining the tools and the input-output schemas around them.

23:49Again, just an example here of a significant investment that you might have poured into making a harness and an agent that is particularly successful with an open-weight model that may suddenly find itself either unsuccessful or providing unreliable results with a closed-weight model because of these differences in integration. One of the biggest impediments that we would face, though, if we didn't have open-weight models tomorrow is that for a closed-weight model to really be a drop-in replacement, it has to remain durably available. We don't have that today in a couple of ways. Firstly, the close-weight providers can change the model availability at any time.

24:28We've had troubles where some of the GPT models were taken out of service and some enterprises had built particular functionality around them that just didn't work with other models. The other durable availability problem is just uptime. Even just this week, there has been significant outages with OpenAI. Thankfully, Claude was still available at the time. Then Claude gets overloaded. Thankfully, OpenAI is available. But you don't face those problems when you have greater control over your destiny because you're using an open-weight model that you are controlling how to distribute, host, and run.

25:03The other problem around durable availability is this thing that I've called out a few times on the show where it feels like, and you know, this is anecdotal and I've seen some people maybe can benchmark it.

25:14James Wilson: But anecdotal or not, it feels unmistakable that when there is a new model coming from Anthropic, the current Opus model is lobotomized. Now, I've had Nicholas Carlini from Anthropic on the show before, and he told me that my assertion that they change the quantization of the model, that is, they take the floating point values that might be a 16-point number and convert them down to an 8-point number to make the model smaller so it can run on less capable hardware. He tells me that they do not do that. They do not quantize the model. It's always the same model. But that just leaves me with the question of, well, Anthropic, if you don't quantize the model, what the heck do you do that seemingly makes it significantly less intelligent and capable in the week or two leading up to a new model happening?

26:01Happened in just the last week or two. I knew that Opus 5 was coming because all of a sudden 4.8 just produced the most just absurdly wrong implementations of things or wouldn't go far enough with tasks that would

26:14James Wilson: otherwise normally complete. That's a real durable availability problem in a couple of different forms. Model availability, outages, quality of the results, etc. But the real one that I want to get down to is that the closed wait API for inference is only a drop-in replacement if the model itself exhibits those same behavioral characteristics that you have spent an immense amount of time ensuring that your agent, your harness, whatever it is that you've wrapped around this LLM, performs just as well, creates the same outcomes, the same high quality results that you or your customers expect with a model that might have entirely different characteristics.

26:56And let me give you some examples here. There is a massive amount of difference between Claude, Opus 5 and Opus 4.8 and GLM 5.2. And I've seen this firsthand over and over again. Disregard the benchmark scores for a second. When I'm working with Opus 4.8 and then later with Opus 5 to do security vulnerability research in macOS and iOS, the Opus models can very confidently and competently identify where a vulnerability is. But they are far too quick to dismiss something as not being exploitable or not being reachable or there being no way to chain a few things together to make it more useful. They just, they stop.

27:41Claude Opus gives up. And again, I can't attribute that to anything other than just a behavioral characteristic. GLM 5.2, on the other hand, when I tell it, you know, here's this other researcher's notes, I'm disappointed that they're saying that this particular vulnerability cannot be exploited. What do you think? honestly it's like cheap glm 5.2 is just like hold my beer jimbo i have ideas and it does it is like this relentless pursuer of other angles other vectors other things it can turn it can chain together i mean i'm looking forward to being able to talk about this more but it turned a what seemed like just a simple arbitrary plist right uh primitive where the file location and the name was a fixed thing and couldn't be changed into actually a race condition-based way to do an arbitrary write of a pay list across the entire file system as root.

28:37Now, that is two very different cybersecurity outcomes. If you're relying on a model that you know can reliably find vulnerabilities, an attacker could turn into a local privilege escalation, and all of a sudden that model goes away and you're left with a model that will stop and tell you that, no, that vulnerability is fine an attacker can't reach it, then you have again incurred a massive impediment and a massive reduction in capability to keep your product safe because of that open weight model no longer being available. These are all the reasons, and I'm sure there's many more, as to just why it's far more complicated than just saying, eh, there's closed weight models available, there's similar API tier costs, there's similar benchmark capabilities, just drop them in and go.

29:22It just doesn't work like that.

29:26James Wilson: But nevertheless, we might just be forced to make it work like that. And how do you get from that's not how it works to this is how we had to make it work? You get there by incurring a whole bunch of cost. Now, this again is where I come back to the notion that benchmarks and token cost comparisons alone are not enough to navigate this situation. You need to build a holistic cost model for your use of AI. and the advice I give us to how to do that is to construct a model where the denominator in this is the verifiable outcomes and then you can start to think about all the costs that goes into achieving just one of those verifiable outcomes to arrive at a cost per verifiable outcome.

30:14So what's that little word soup mean there? So the reason I think this is important is if you just compare token budget before to token budget afterwards, you're not taking into account the quite substantial costs outside of the token spend itself. For example, how much additional human effort is needed, either ongoing because of a different model behavior, or as an upfront investment to retool, to make adjustments to harnesses, to deal with differences between the open-weight model you were relying on and the closed-weight model that you might be forced to switch to. There could be things like, you know, one model might reach an outcome pretty quickly with limited reasoning, but for whatever reason, the other model that you're forced to move to might rely on far greater reasoning.

31:01And so whilst it may look like it's similar sort of output from a model transcript perspective, there's actually a massive difference in token cost to just reach that same verifiable outcome. And then you've got the nature of, because we're talking about a verifiable outcome, you've got, assumably, a fair degree of human involvement to do that actual verification. I don't think we're quite at the point where we completely can rely end-to-end on a model achieving a cybersecurity outcome like finding a vulnerability or an exploit, and then the model fully validating end-to-end that, yes, that is a complete description of the vulnerability in the sense where it's good enough to then be reported to a maintainer or a corporate bug bounty or whoever you're reporting these vulnerabilities to.

31:48The other reason that a verifiable outcome is important as your denominator for your cost model is it can take multiple attempts through an agent harness or an iteration to achieve that outcome reliably. You know, let's say, for example, you've got a particular harness setup that you know is able to find a specific bug in an existing code base, but you find that it takes, let's say, 10 iterations for that harness to find that vulnerability once. If you accounted for the cost just on that successful run, then you'd be missing the fact that it took 10 times the cost to actually find that verifiable outcome, the full encapsulation of the overall cost.

32:28And this could become the thing that becomes quite variable in the mode where you have to switch from an open-weight model to a closed-weight model. It might be initially that the initial cost you incur is a significant number of additional iterations of the model to make up for the fact that this model doesn't respond as well to the inputs that you're providing it or the way that your harness operates. Building a cost model like this to be able to baseline your total cost per verifiable outcome is essential for you to be able to both proactively model out how is that likely to change if you had to switch to a closed-weight model.

33:04You can look across all the costs there and make a judgment call as to what's the probability that there's a significant increase in spend or maybe decrease in spend because of the different properties that we've been talking about between an open-weight model and a closed-weight model. It's also very useful to just keep a track of the realization of those changes in cost model to be able to answer the question of how much did the switch to a closed-weight model cost you. But the other flip side of cost, of course, is who ends up paying for it? The answer is the US operator first and foremost pays the cost.

33:41That cost gets passed on to customers. And for a move that seems to be aimed at strengthening the AI economy in the US, it really doesn't seem to stack up to doing that, at least in the initial phase of this. And I don't believe that a frontier laboratory lab benefits in some linear fashion whereby the spend that was going to people that were hosting a Chinese open-weight model suddenly becomes just a one-for-one redirection of spend to the US models. Because I think for all the reasons we've discussed here, the additional cost that is going to be incurred in facilitating that transition will actually be spent elsewhere.

34:18And I think there's a pretty strong case that the actual spend with these US-based frontier labs is the expense that enterprises try to continue to minimize and minimize and minimize through spends in other directions. And just before we move on from cost, the other thing to remember here is that not only would a restriction on the use of open-weight models drive a huge amount of cost that is borne by the cybersecurity operator and then the downstream customers of that operator, but it also continues to create an asymmetric advantage for an attacker versus a defender. The reason for this is there's one other cost here we haven't talked about, which is opportunity cost.

35:00If we were faced with the predicament of no longer being able to use an open-weight model and having to spend the time to do whatever's necessary to make a closed-weight API as inference model work for us, then honestly, while we're doing that as defenders, the attackers are uninhibited and continue to ramp up their skills, further integrate AI, and get even better at mounting larger scale attacks with even higher success rates. So it is just, it is such a frustrating thing to consider because it just does not seem to result in any of the outcomes that such a move would reportedly be aiming at achieving.

35:40James Wilson: But as we wrap up now, I want to zoom out and ensure we have a balanced perspective. As I said towards the start of this podcast, I can honestly steel man in my mind the case of why a policymaker might think that it is tractable to enact a regulation that makes it difficult or commercially non-viable, restricted for a US entity to use these Chinese open-weight models and or to impede the creation of new Chinese open-weight models. because at face value, when you compare the closed weight offerings to the open weight offerings, there does seem to be comparable capabilities and price points. But for all the reasons we've just mentioned, there's far, far more to consider.

36:25But again, in the spirit of taking a balanced perspective here, my point of view here is not that regulation is bad and all regulation is going to fail, but rather my point here is that the macro effect of regulation that restricts access to open weight models and or prevents the creation of them, ultimately

36:44James Wilson: regulates the United States out of the innovation feedback loop that is centered around open weight models, regardless of their origin. And this is because an open weight model is not a finished product. That model can then be made even more useful over time through fine tuning, distillation, quantization, all these incredible techniques that are just not possible when we're reliant on closed-weight models that are behind APIs. Open-weight models let researchers and startup and operators turn one laboratory's release into many specialized systems that get better and better over time. And, you know, a U.S.

37:21ban, well, maybe it won't erase the already released model weights, but it absolutely will prevent compliant Americans from making them safer, cheaper, and more useful because U.S. researchers stopped producing and evaluating derivatives of the prohibited models. Defenders would stop discovering weaknesses in and contributing fixes to not just the models, but the immense amount of tooling and supply chain that goes into, again, building the car around the engine block that is the open-weight large language model.

37:52James Wilson: It would also have a lot of impacts on startups and hosting and specialized expertise. All of that moves away from jurisdictions where they can't participate in this open-weight flywheel of innovation towards jurisdictions where that remains lawful. Ultimately, foreign operators continue to learn from the ecosystem, while compliant US operators would not. And look, that may be a trade worth making. The restrictions may buy time and reduce misuse or deny revenue to these Chinese open-weight model labs that have been accused of IP theft and distillation. And perhaps that is worth the macro effect of the US being shut out of the open-weight innovation flywheel and the very specific impacts to cybersecurity defense.

38:37And, you know, I'd be honest here, I don't have the expertise in this macro and policy

38:44James Wilson: and intelligence world to really offer an opinion there. And maybe I'm being naive here, but what my expertise does lead me to is a strong belief that it would be much better for the US to participate in making safer, better open-weight models regardless of their origin rather than impeding those efforts while simultaneously harming cybersecurity defenses and overall AI innovation in the US. I've been James Wilson for Risky Business Features.