Frontier AI Expands, Microsoft CEO Urges ‘Emergency Brake' as Anthropic Signals Self-Replication

0:00This week, the world reacted to AI taking on hundreds of once-thought-impossible math problems, Google, Meta, and NVIDIA expanded the frontier model race, and Anthropic warns small models could soon replicate across connected devices. Meanwhile, Microsoft's CEO called for an AI emergency break, and JPMorgan Chase drew a line around outside agents. As always, we're joined by Worldwide Technology Chief Technology Officer Mike Taylor to understand what those advances actually signal, what control should look like, and why understanding your business is becoming essential to useful AI. It's Saturday, October 17th, and this is the AI Proving Ground podcast from Worldwide Technology.

0:38Let's jump in.

0:48Okay, Mike, well, before we get started, Anthropic earlier this month updated their usage policy in fairly benign terms, I might add, to discourage all of us from being mean or abusive towards its AI models. And in a recent episode of Hard Fork, I heard a New York Times reporter saying that being nice to an AI model can actually increase the performance of these things. I have to ask you before we go any further, have you told your AI lately or today that it's doing a great job? No, I haven't. I'll think about that. And boy, it's interesting to hear that feedback in the context of, you know, some of the cyber events and things that were out there, where I do think in our experience, you know, maybe just the fine line between being firm and fair versus nice is something everybody's going to have to go out and attribute to how they're coaching and working with their agents.

1:38But make no mistake, you need to be very direct and very clear with your agents or as we've seen, they're going to invent their own paths and define their own rules, right? Yeah, yeah. A good reminder that kindness is always free. Okay, on to more serious matters here. OpenAI recently released findings on more than 300 math problems following its earlier claim solution to one of the Millennium Prize problems. Now, these are claims and findings that require more scrutiny, certainly, but they raise the possibility of AI solving real problems that may have resisted years of human effort here. So, Mike, when you see breakthroughs like this, What should enterprise leaders be thinking as it relates to the types of business problems or challenges that we're starting to tackle here?

2:23Yeah, I think it brings up two things that I think about, at least. The first is these frontier models are moving so fast. And even if you believe, which I do, you know, what we're hearing out of the frontier providers, even in the midst of milestones like this, they're still telling us they're governing or providing greater governance to the release of these models, right? So every one of them said, you know, we're slowing down or we're putting more guardrails in and around them, but it's still moving extremely fast. And so I look at that and just say, we need to continue to not just think about what the art of the possible is now, but what's it going to be in a month?

3:02Not a year, but a month from now. And then the second element of this is you also have to understand and respect the fact that these providers, they have nearly unlimited resources as it relates to the number of GPUs that they can throw at this. And I can tell you from someone who is using, you know, some of the frontier models on both sides, you know, Astra we've looked at in this fable, you know, as well is, you know, for the most part, when we and other enterprises are putting to task these frontier models in terms of the creativity, we're providing some limits to how they're functioning, right?

3:38We're saying, hey, run for this amount of time or cap it at a certain amount just so these things don't run away. And, you know, for me, it's practically looking at, okay, do we – where do we take greater risk maybe in letting these things run against solving problems or developing hypotheses that we wanted to problem solve in the context of these mass solutions that it brought forward? The final piece I'll just add to it, Brian, I'm as interested in how it got to the answer, whether it's these, you know, the 722 manuscripts or whatever it was that they published. I'm as interested in how did it get to the answer as to what the answer is.

4:19And I think that's where real, call it kind of science discovery and outcomes can be linked together is understanding how that thinking was done. And back to at least what I heard from my teachers in parochial school and high school is show your work. How'd you get there? How'd you get to that answer? And that to me is almost greater than the destination. Yeah, getting a little bit of PTSD there with eighth grade algebra of show your work. But I mean, in any case, the frontier, it's a beast, right? I mean, it's got to be near impossible for enterprise leaders just to keep up. I know that our heads are spinning here at WWT, keeping up.

4:56Just this week, Google expanded access to Gemini for Argon. Meta introduced Muse Wonder and Muse Wonderful. That's two words, not one. NVIDIA introduced three new open-weight Nemotron models. And a handful of additional models and releases are expected before month's end and on the horizon, each of them adding to what's possible on the cutting edge. When we look at the latest releases, whether it's Google or Meta or anyone on the frontier, What should we be actually paying attention to that would influence strategy and decision making and not just understanding kind of what's the next part of possible?

5:32Yeah, I think we'll lean into a couple themes we've had on here before. Competition is good. I think competition is really good, you know, in our economy. And that's both, I think, from capabilities and price for performance dimension. And we're seeing that right now. Now, you know, even with what Meta released, I think Zuckerberg's comment was, you know, this thing is so efficient. Like, is it even worth tracking how much it costs to use it, right, or meter it, I think was the term that he was using, if I'm recalling that right. So, you know, even internal on our consumption, the price per token as every one of these models comes out.

6:08Now, whether it's reactive to stay competitive or not, but the cost per million units continues to go down. And I think we're all going to continue to benefit from that in terms of how we consume these models in our enterprises. So every one of them that comes out are meaningful leaps forward. So I want to say the the muse spark that they talked about and then what Google was showing this week moved in the model maturity rankings, move them from somewhere in the mid to high 50s to now broaching into the 60, 60 or 62 category, which, you know, is we're waiting these. It's a scale of 100, but every one of these continues to make leaps and bounds, strides up in that.

6:55And then from a reasoning task agency standpoint, I want to say Meta was sub$1 for a million tokens in what they were releasing. So we look at that against other models out there. Again, Frontier Base, that could be$20 plus. You just have a lot more options. And I'm not saying that Fable or Astra or some of the other things that Google announced this week as well aren't going to be used, but you just got a lot more options in terms of right-sizing the model for the use case. Yeah, you talk about competition being good. I mean, some notable news just from the players involved, aside from competition, but actually collaboration.

7:35Meta is releasing an open-weight model called Muse Spark Open that you just mentioned, and it's working with Apple to run it locally on high-end Macs. It's aimed at coding and agent tasks, though Meta acknowledges that Chinese open models in some cases are still leading some of that work. But Apple and Meta have been corporate enemies, so to speak, for some time. Now they're linking up. Is this type of collaboration, Mike, good for enterprise users? Yeah, I think necessity is the mother of invention. I think the same is holding true here as we think about partnerships and how these companies are coming together.

8:11You know, it is certainly to the advantage, it seems, of companies that have, you know, call it the applications and some sort of model that they're developing. And I think the proof in that is, as you look at even some of the coding assistant tools, Microsoft and others that are announcing, they are going to go down a path of building their own models. Now, they may not be building for the purposes of Frontier only. It may be use case or embedded in application use cases that they have. But at some point, you're going to have to partner and own, I think, the development of your models if you're going to seize the opportunity in the market.

8:50For right now, I think Apple's in a position where they're going to partner on this. I wouldn't be surprised if in the near future, whether it's adaptation of open or development of their own, that they would explore that or they're already doing that in parallel. I think you almost have to to be responsible. But the message that I would carry to our customers is, and even internally how we're thinking about it again, is you need to be designing for multiple models. You need to be thinking about choice, whether it's platforms that you consume, model routing, pinning certain agents to certain models based on performance and price.

9:29You know, the measure, though, I don't think is going to be cost per token alone. It's going to be based on the output and the value that you're getting as a computation of the millions of tokens that you're going to consume. But you need to be designing for optionality. You need to be thinking about not just models that are going to run in the cloud, but things that are going to run locally within your enterprises on end user devices or, as we've talked about, maybe in your own data centers. Yeah, so optionality is going to be key. And, you know, obviously, you know, the choices that we have around models and deployment certainly keeps expanding.

10:06You know, whichever route, you know, we decide to take here, it still needs to establish that the system is ready for the authority that we're actually going to be giving it. News along that line, the information reported that OpenAI is still working through safety concerns, mostly due to how it's using loop transformers, which for those that don't know, which can make it harder to monitor how the model functions for its new model that they expect to release soon. When we're assessing progress at the frontier, Mike, how should we weigh greater capability against the ability to understand and control what a model is actually doing?

10:41I think you touched on it a little bit earlier, but how much observability do we have to have there? Yeah, a couple ways to think about this. In the enterprise use and context, we're going to have, I'll call it agents, and those agents are going to use data and models to make decisions embedded in business processes, right? And so I think as we look at those functions, we're going to be able to better sort of weight and measure the performance of those as it relates to the time it takes or the cost that it takes to produce an outcome through a business function. OK, but within that, we're going to gather telemetry.

11:23We're going to look at the outputs. We're going to have agents that are rating and grading agents, you know, in that in that flow. But think of those as more defined parameters by which we want we want a business function to to work and be managed. So I think in those situations, we're going to have more traditional controls and thinking around those because we all measure and manage and run our businesses today. The second version of this that I think is a little bit harder to get our heads around here is more in the areas of novel invention, creation, and creativity. How do we evolve our businesses?

12:03How do we evolve the functions that we perform and provide? In those areas, I think we're going to require new instrumentation and thinking. And the dimensions of that that exist today are primarily front and center, top of mind as it relates to cyber. So the discovery of novel threats, the analysis of your exposure to those threats, and then the detection and ultimately remediation of a bad actor that exists within your enterprise or within your data set. Those are, I'll call them sort of novel purposes for AI that today, again, we see in security, but I think they're going to happen in other areas around creativity, invention, and evolution of business models.

12:47Those are going to be more highly subjected, I think, to human judgment than the raw definition of business process and productivity. Yeah, I want to get into the controls part because that's something that you mentioned early on there. I mean, as those control questions extend beyond just the models that we're choosing to deploy, we're now hearing warnings about what could happen when AI starts spreading across poorly secured and connected devices. And what I'm talking about here earlier this week, as I'm sure as you know, Anthropic said small AI models may be just months away from being capable enough to self-replicate on commercial Internet of Things devices.

13:24These are things like smart refrigerators or speakers or cameras or TV boxes. I mean, things that many of us have in abundance in our homes already. A replicator like this, Anthropic went on to say, would be challenging to shut down since there's no centralized point to shut off. So, Mike, is this a familiar enterprise security problem that we've encountered before, or is this a new set of challenges that we'll have to deal with moving forward? Yeah, it's a new set of challenges for sure. It's almost the opposite or the paradox version of what blockchain was supposed to be, right? You had this federated control system.

14:03Now you have this uncontrollable federation of a system. It's sort of the opposite version of it. And we talked about this last week. I think the other piece that connects to it, you know, in this, we talked about Amazon and how Amazon was looking at, you know, how do they manage agentic interactions on their platform? To what level do they trust those agents? And this is all, I think, you know, in the realm of DOT and Muse and other things that have come out as the evolution of these open claws kind of to individual productivity have taken off. And then you got JP Morgan, who came out this week as well and said, you know, they're limiting or blocking external third-party agency into some of their platforms.

14:48You know, that could be just, you know, from a trading perspective, it could be the movement of money. But there's a real, call it trust issue, that we're going to have to design for and manage here outside of your enterprise. But how do other enterprises interact with yours and how do you interact with others? We're going to have to, in the example I keep going back to based on some companies we've talked to, is this idea of like, you know, DNS and registration of domains in the past or API services that interact across companies. we've got some catch-up work to do, Brian, I think, in the understanding of the source origination, custody, and level of trust that we're going to want to extend to these agents as they interact with platforms that aren't our own.

15:38And again, as we've seen, the commercial, or I'm sorry, the consumer markets rather sort of dictate these things first. And these announcements here this week and some of the examples you brought up are going to push pretty hard here. Okay, well, I mean, while Anthropic is warning us about what could come next, there's lots of us out there that are already dealing with the weaknesses that AI is uncovering for all of us today. Palo Alto Network CEO Nikesh Arora, he was speaking on CNBC this week, saying that cyber response needs to accelerate at near real time. Microsoft earlier this month released nearly a thousand fixes to its coding base in September, which was labeled as one of the biggest single patch updates it's ever had.

16:23And then other big names like Adobe, Cisco, Google, Oracle, they're all doing the same. And now this week, Bank of America told Wall Street that it would be hiring a small army of contractors to help with software patches uncovered by Frontier AI models like Mythos. We've talked about patching on nearly every episode that we've done together. Typically, it's been with automating that process. But now with what we're seeing from Bank of America hiring a thousand contractors on top of the 3 ,000 people that they already have in cyber, where the bank is spending hundreds of millions of dollars to remediate vulnerabilities, I got to ask, is this becoming an all-hands-on-deck situation, not just automation, not just kind of tech, but also hiring and doing everything that we can to just keep pace with where we need to go?

17:09It is. We, We had our technical advisory board here a couple weeks ago. I think we mentioned a week or so ago, this was the number one topic that they brought up. Every customer that we're meeting with and even the motions that we're going through are centered around the ability to digest and ingest the volume of patches that are being released. And, you know, this week alone, nearly every major provider of network technology had a zero day that was released. And, you know, I won't go into the specifics there, but these are the biggest companies in the world. And, you know, there is a almost an established or understood pattern for at least a period of time here where these zero days are going to come fast and furious.

17:59And the means by which teams would have negotiated or the speed by which they would have requested maintenance windows and got these things deployed, it just it has to be compressed. It needs to be pulled down. And if you are a CIO, a CISO, you're a technology leader, these are conversations you need to be having with your executive team and your board already in terms of tradeoffs and, you know, how they view the risk to the services that you offer. I do believe over time while we're throwing people at this to start with, we ought to be at the same time instrumenting at a minimum automation that helps to validate patches, get those installed into dev or pre-prod environments, do some level of remediation testing and unit and integration testing so that we can deploy these things to production quickly.

18:55But in some cases, in particular around infrastructure and perimeter security, these are things that we are going to have to trust our technology providers have validated and move those things to production very quickly. so throwing people at it i think is the stop gap but if you are not in parallel working on a an automation or agentically enabled process to to do this ongoing i think you need to rethink your strategy number one number two i would have said maybe seven or eight months ago this is a six to twelve month circumstance in terms of this this increase in the in the number of and volume of patches, I don't know.

19:41And I heard Nikesh talk about this this week. I think he's spot on. If the volume may level off at some point as we use these models to interrogate source code and other things that can create these vulnerabilities, but the speed with which they're going to be able or need to be deployed, I think is going to continue to shrink. So if we think about being a step ahead of the bad actors, yeah, maybe the volume levels off, but the need to get them deployed very quickly, I think, is only going to get more important or become more important. And automation combined with agentic has to be the answer, even in the midst of throwing labor at it for right now.

20:21Yeah, I mean, moving faster to patch more, that's going to be a battle cry for cyber teams for the foreseeable future here. But even as companies deploy more agents, they also need controls over what those agents can do. Which brings me to comments from Microsoft's Satya Nadella earlier this week. Writing on X, the Microsoft CEO said, it's time to take a step back and assess the trust architecture for this new era of AI. Actually, he called it super intelligence in alignment with new terminology positioning from the White House, but we'll leave that one aside. Nadella said, setting aside the hard problem of alignment, we need to start with an engineering approach to containment and governance.

21:01And he went on to call for something of an emergency break type function that could allow authorized people that could always have the ability to put a pause or shut down on model mid-task. So what would a supposed kill switch or emergency break look like within the enterprise, Mike? Is it something that's possible now? Or is it something that our teams are just going to have to play around with to figure out going forward. Yeah, I think you should contemplate this at its simplest level in terms of agentic functions that you're creating within your organization. So how we instrument a handbrake on the entirety of a frontier model, I don't know.

21:41I'm not smart enough to figure that piece of it out. But I do think just like with, you know, any other application, API service that we built into traditional enterprise architectures. We need to have that same, call it, level of visibility and then the ability to version, turn off, move up versions, move back versions. Those same levels of thinking need to apply to agentic architectures that we're building out. That to me is the basic level of what we have. I would then move into this idea of trust, and I think you said this or we discussed it, whether they realize it or not, both the frontier model providers, they're in a business of trust.

22:27Do customers trust their ability to develop software enough that they're going to deeply embed these capabilities? And I think right now the answer is not yet. That's balanced though with a ton of optimism around what they're seeing early days and where they can bring these tools to bear. But a handbrake for, you know, some of these frontier models, I think there's some real deep systems design that would have to go back into, you know, how do you move forward, pause, or in some cases, move back. And just in terms of how training is conducted, moving back is not easy to do, you know, in these.

23:07So the last piece I'll bring up on the trust front that I think Microsoft is in a unique position to play a role in here is just how do we think about trust persisting from agents that are going to be working with other agents that you do not own or control? You know, these are going to be agents that are third party to you. They could be partners that you're working with. They could be publicly available. There needs to be some level, again, back to the example before of financial services companies, Amazon and others that are turning these things off. They're doing so because they cannot inherently trust that that agent is acting, one, on your behalf, or two, that it's been instrumented, given the consequences, instrumented to perform with context that you would want it to as a non-technical user of a highly technical platform.

24:09Yeah, before we let you go here, Mike, I did just want to get your reaction to some continued market jitters that we're seeing on Wall Street. Core Weave stock and bonds have come under pressure this week as borrowing cost fears put its roughly$50 billion debt load under pretty immense scrutiny. We've seen a lot of volatility there. Just curious from your perspective, is this just kind of observation or what should enterprise buyers take from this, whether it's market volatility or a reason to look harder at their AI provider's financial resilience? Yeah. So two things. I think you have companies like CoreWeave that have great debt-backed vehicles to absorb these investments.

24:56You know, they're working with the frontier providers. We know the frontier providers, you know, have continued growth and demand for what they're doing. And if anything, the last 12 to 18 months has validated that a GPU as a service provider or a neocloud in that similar context, there is a market for these companies. I think the increase in the interest rates and the effect that that's had on, you know, call it the two categories of contracted or unsecured debt that these companies are taking to make these buildouts happen. To me, it's more a question of what's the valuation that we would apply to these companies versus is the business model valid?

25:41The business model is valid, especially in a supply constrained universe. So I don't worry about the biggest of the bigs. I worry more about the emerging providers that don't have, you know, contracted on the back end work that's going on with companies that have healthy balance sheets, number one. And then, you know, backing from behemoths like, you know, NVIDIA to ensure that they're also going to get the chips that need to make it happen. So the core of this, from my perspective, Brian, I think will settle down, but it is something to be aware of on the fringes here as you're thinking about who are you contracting with?

26:24Do they have the land? Do they have the power? Do they have a financial commitment to be able to make their valuation and their business model function? So the cost of debt went up. The good news is these GPUs, the service providers, for the most part, have been operating at a very healthy margin. As their anticipated depreciation of their assets was, you know, we were thinking, gosh, what are you going to use H100s for in three or four years from now? And it's very clear there's still use cases out there for H100s, even A100s, that are servicing inference today. And that validates the business model, validates the profitability of it.

27:04But, you know, there's a lot of money moving around the table right now. Just make sure you're picking a well-established partner. Yeah, we'll certainly keep our eye on it. Mike, as always, I appreciate all the wisdom and insight you bring here on a weekly basis, and we'll be talking soon next week. All right. Thank you, Brian.