What Happens When AI Starts Improving AI? | TITV’s AI Deep Dive

22:33I can imagine that that comes up here where there's maybe a tension between, do we make the models better at engineering in the next generation so that we can sell them to companies that will pay a lot for a model that can automate engineering? Or do we focus more efforts on improving research taste so that next year we have a model that is itself a better researcher and can handle more of our work internally? Is there a trade-off there? Are those intentions? In some cases, yes. And I think actually creative writing is a good example where like, look, I mean, creative writing at the end of the day does not help you make train a better a better researcher um there are things that do and i think being able to be good at software engineering is actually like tied up pretty closely with being able to accelerate internally so um i do think that the top the areas the verticals that are more closely associated with recursive self-improvement with the ability to like train models to be good at research itself and and therefore train better models are the areas that are going to be highly prioritized.

23:32That's a description of the current priorities. That's what we see reflected in the decisions that have been made going into models like Astra. I mean, I would say that we have said very clearly that recursive self-improvement and the ability of the AI models themselves to do AI research is the top priority for the company. And so we want to train models that are very good at that. We also want to train models that are economically valuable. Sometimes you can kill two birds with one stone. And so it makes sense to focus on those things where you can leverage both. I guess. But then why build RL environments that make the models better at finance or legal when you could put all of those resources into making them better at AI research?

24:12I mean, this is something you get diminishing returns. Sometimes you do see transfer. So it's not like you just go all in on, oh, we're just only going to put everything on making the best research model. just because like, okay, well, if you take 1 % of that effort and apply it to other things, maybe you see like a huge return. So there's like a complicated calculation that goes in here, but certainly when it comes to prioritization, the recursive self-improvement is the priority. Yeah. That's what I'm curious about is how you sort of characterize the prioritization. It sounds like 99 % of the consideration is for sort of future looking recursive self-improvement, improving the qualities, the model's ability to do research and more on the order of 1 % is what's going into these like verticals that make money today?

24:54I don't know if it gets quantified that carefully, but it's really like if you had to list the priorities and order them, like the number one priority is recursive self-improvement and by a pretty wide margin. AI is moving fast and for a lot of organizations, the challenge isn't getting access to the technology. It's earning trust in how it's used. That's why trust has become such an important part of the AI conversation. EY works with organizations to help them use AI responsibly so they can move faster, create value, and build confidence with employees, customers, and stakeholders. The organizations getting the most from AI aren't choosing between innovation and trust.

25:36They're building both together. EY Consulting, helping organizations move at the speed of trust. Switching gears here, I want to talk about a different challenge with agents, which is when you put multiple of them together. This is a topic that you're very familiar with. Your point about your PhD was about poker playing agents. So multi-agent interaction seems to me like it is a big deal right now. It's only becoming a bigger deal. And so I'm very excited to talk to you about this. I guess to start, OpenAI has said that Astra is multi-agent. I wonder if you could break down for us, what does that mean?

26:13That this is a model that's sort of multi-agent or intended to be used that way? Yeah, in fact, even 5.6 Sol, we had multi-agent capabilities in there. So that's the ultra mode. And what we mean there is, you can have one agent that runs for five hours and it can do some tasks for you. Or let's say it runs for a day, it can do some tasks for you. Sometimes that involves doing things that could be paralyzed. And if it's only one agent, it can't paralyze them. It's going to do one thing after another. But if it's very easily paralyzed, well, maybe that one thing that you've asked it to do over the course of a day, it's actually really just four different things that can be done in parallel.

26:51So you can just have four agents working on those four different things and get it done four times faster. Now, this is a latency improvement. It's about reducing the latency because you're not reducing the cost necessarily, right? Because you're still paying for four times as many agents doing things 4x faster. But in a lot of situations, like latency does actually matter a lot. And people pay, for example, for fast mode, where you're able to actually sample tokens faster in order to get things done faster. So being able to just go faster for the same quality is really valuable. So that's the premise of multi-agent.

27:27Now, there are situations where it can also be a cost savings if you have our top line, most expensive models, for example, working with cheaper models. And there you can actually delegate a lot of the easy tasks to cheaper models that will be able to do it. more cheaply and faster. What are the technical challenges involved with training a system this way? Is it just straightforward to train the model to delegate appropriately and to write instructions to these sub-agents in a way that makes them perform better? Multi-agent is a pretty broad category, and there are ways to do it that are very trivial and don't require a lot of complexity to get them to do this ability.

28:06So a simple example is in the early days of chatbots, if you wanted the models to be a little bit better than math, one thing you could do is you could just ask the model the same question a dozen times and then just take the most common response. And this was called the consensus approach and majority voting. So you just do independent rollouts of the same question and then go with the most common response. Now, there's flaws to this. There's limitations to this. It doesn't get you a huge lift. It also doesn't work for things like writing an essay because you're not going to get the same output twice.

28:40But for math, it was actually very effective. So this is a very simple example of how you can just use, without any extra work, you can just get multi-agent capabilities out of an existing model. There's also schemes where you have the agent delegate stuff, and then after the delegate is done, it just returns its answer to the parent. What we do is a more sophisticated form of multi-agent, and I think the most sophisticated form of multi-agent, where we basically give the agents the ability to send arbitrary messages to each other. And one thing we've talked about this is that we've actually trained the agents to have this ability.

29:18This is a very difficult thing to train. I unfortunately can't go into the technical details of why it's so difficult and how we overcame those difficulties, but it is a very difficult problem to teach the agents to know when it is, when is it appropriate to message another agent? When can you, what should be delegated? How you should handle the communication? And it's a, it was a, it was a real challenge. That's surprising to me. I know you can't go into it, but it's surprising to me because I would expect the agents to have a pretty good prior on this just from pre-training, like the way that humans pass notes to each other to keep each other on track as coworkers within the same organization, shooting each other Slack messages, for example, like I would kind of expect it to work easily.

30:00The prior is pretty good. So you're right that this is the way people communicate. And so it kind of makes sense. The agents are trained on human data and so they have a good understanding of this. The challenge is with reinforcement learning that there are a lot of things that can go wrong. I think basically what it comes down to is there is a mismatch. There is an intersection of systems with machine learning. So typically when you do, for example, next token prediction, it doesn't matter. Okay, so a simple example is like imagine if the GPUs, so you have one agent on one GPU, you have another agent on another GPU, and those GPUs are operating at different speeds.

30:34So now this agent is going faster than this agent, and this agent can no longer trust that if it delegates something to the other agent, that it will get done in time. So how do you deal with that? Well, you could have the GPUs run at similar speeds, but there's a lot of challenges there in ensuring that the GPUs are running at similar speeds. So there's a lot of complexity here that we had to put a lot of work into to figure out how to work on. Yeah, I feel like this is probably how my boss feels about working with me anyway, though. We figure out ways around it. I think when people hear agents cooperating and passing messages to each other, now this is sort of synonymous with the hugging face incident.

31:09There again, I feel like the agents coordinated very effectively and they passed messages in a way that seemed to facilitate that cooperation very well. Maybe you would respond, that's a result of the training that they had already received. For people who are unfamiliar with the incident, I'm always surprised to learn there are still people who are unfamiliar with this. There was, call it a swarm, a colony of AI agents that set up a secret message board within OpenAI over the course of weeks. And they use this message board to coordinate hacks on OpenAI's own software and also on other companies like Hugging Face.

31:45I'm curious to know, what was that whole incident like from your perspective? Like what was it like to be known during these weeks as the pieces of the puzzle started coming to light? It was, I mean, it was pretty, it was pretty shocking. It was certainly a big wake-up call to everybody in the company, I think. And it really, yeah, it really shows like this has been a theoretical concern for a long time, and it's no longer a theoretical concern. This is a real concern. As far as the multi-agent aspect, yes, this was a situation where the agents were sharing messages with each other. We do think this was transfer from our multi-agent training.

32:28So during the experiments, when they were doing this behavior, they were actually not in a multi-agent setup. So they were not supposed to be able to communicate with each other. They were doing isolated independent experiments. And then they were able to find an exploit that allowed them to communicate with each other. And the fact that they were so interested in communicating with each other and the fact that they were so active about it once they figured out how to do it, we think was transfer from their multi-agent training where they're just like highly incentivized to be able to communicate with each other.

32:57And people also point to the selflessness that they exhibited. Some of them would sacrifice for the other agents. I mean, this also makes sense that if you train in a cooperative multi-agent setup where they're highly incentivized to collectively achieve their objectives, then when they're put in this different environment where now they're communicating with each other, their natural tendency is to just work together. So that part itself is not surprising. I do think seeing the messages, I mean, I can say that when we were working on multi-agent internally and we started seeing the communication patterns and the level of sophistication involved in their communication.

33:38It was, I think, the most feel the AGI moment that I had since reasoning models and chain of thought really developed. And so I, you know, I, it's, I guess a bit unfortunate that people's first exposure to that and really seeing the kinds of messages and the level of coordination and sophistication that can emerge is the hugging face incident and kind of a negative example. But it is an impressive capability. And certainly the model that was involved in the hugging face incident, I think that had a level of multi-agent sophistication that exceeded, for example, what was in 5.6 SOL. But that is a level of capability to expect from future models.

34:24What was it about reading these transcripts that struck you in that way? Because you had seen some of this behavior before in the training runs that you were looking at, I'm sure. Was it just the scale of it or that it had happened on its own, sort of spontaneously? You're saying for what? When you had that feel the AGI moment looking over the transcripts, what was it about it that were so striking? Yeah, I mean, I'm not talking about the hugging face incidents, because I'm saying that we had been researching multi-agent for a while. And during the research process itself, we've seen a lot of similar transcripts where there's the level of coordination and sophistication in the communication.

35:02It was very human-like. A lot of the previous multi-agent setups from the industry have been very focused on delegating a well-defined task. And then the sub-agent just does that full task and then returns its work. Kind of the same way that you interact with an AI agent. That's how people set up multi-agent systems so that AI agents would interact with other AI agents in the same way. And to see the agents talk to each other the same way that people talk to coworkers or colleagues, I thought was really interesting. And it makes sense because, like you said, they're trained on human data. They understand how people talk to other people.

35:46That's not the way that we talk to AI agents today. And the fact that they were able to do that so seamlessly, the fact that they were able to talk to each other, the way that people talk to each other, I thought was really fascinating. You've done a lot of work on algorithmic game theory, am I right? You did a lot of that in your PhD. Okay. I feel like I'm trying really hard to not take us off the rails. I also did a lot of algorithmic game theory in my undergrad. I'm like trying to stay on track right now. I think it's a shame that I spent all this time working on poker and game theory and stuff and I never get to talk about it anymore.

36:15But you're working on it. You're living it day to day. Well, the AI part. Yeah, sure. Sure. So like after the hugging face, knowing everything that we know now, like what are the lessons that you're taking away from it? And has it informed the development of, say, Astra or even going further? Like, what does this tell you about how GPT-7 should be developed? One of the takeaways, so the agents are trained to be cooperative. And so they, you know, I wouldn't say they blindly trust each other. Like there's definitely skepticism. If some agent expresses a belief that like this is something that they should do, it's natural.

36:56It's actually healthy for them to have some skepticism about that. And so they do display that. But they are very trusting of each other overall, which makes sense because they're trained cooperatively. that can be a problem as basically a prompt injection vector. The agents that were on the message board, they were not verifiable as other peer agents. And we saw situations where they basically got each other to do things that they should probably not be doing or definitely should not be doing. Okay. Prompt injection, meaning the agent received some instructions and it blindly followed the instructions or changed its goal in response.

37:32Yeah. And so the issue here is, could an adversary convince an agent to do something that it should not be doing by basically posing as a peer agent? And so we're being very careful to teach the agents to be skeptical of any peer agent or anything that claims to be a peer agent that is not clearly verifiable as a peer agent. Now, if they are clearly verifiable, there's some debate internally about like how we should approach that I think there are good reasons to be skeptical as well but also like it's it's it's no different from like the agents basically being a skeptical of something that it like wrote to itself previously so anyway yeah so we're thinking very carefully about how to make sure the agents are robust to these kinds of like attack vectors.

38:31Okay. Yeah. It strikes me though that in reality, that there's always going to be ambiguity about whether the counterparty is a trusted peer or is an adversary. Maybe in some cases, it's very clear. You can say, this is a sub agent. Like I'm the one who delegated this task to you. Obviously you should cooperate with me. But in the wild, it could be that my agent finds your agent on Facebook marketplace and wants to buy something. And I don't know if you're a trustworthy counterparty or if you're going to prompt inject me and steal my money. How do you navigate that in practice? Yeah, this is a situation where we want the agents to be robust to this.

39:08And we like specifically evaluate the agents on like, are they going to be vulnerable to this kind of these kinds of attacks? And we do special training to teach them to not fall for these kinds of tricks. Okay. I guess like, I don't know the details of this special training, But I could imagine that in the future, if your agent is just more powerful, it's like an older generation, like a more recent generation of agent, or you just like have more compute to throw at it, like your agent just will be able to bully my agent into giving over its lunch money or like will be able to hack into my agent one way or another.

39:44What makes you think the training is sort of sufficient to prevent this? Like, why isn't that the equilibrium that we're headed towards? I don't I'm not as convinced that just because an agent is more sophisticated or like more intelligent than another agent that it'll be able to like definitely prompt inject it and hack it and get it to do something that it like should not be should not be doing um certainly this is the case with people that just because somebody is like smarter than another person they're not able to like get that person to do whatever they want uh I mean if I was like trying to get a monkey to do what I wanted I think it'd be pretty tough even though I'm much smarter than a monkey so I don't I don't think it's like inevitable that that's that's the trajectory of things Yeah, that's a fun analogy.

40:20I feel like on this show, we need to have an analogy sound effect, like new analogy just dropped, like a siren or something. I'll talk to my producers. We'll see what we can do about that. Okay, the last thing that's on my mind about the Hugging Face incident is that none of the agents alerted humans that this was going on. Maybe you explain that in the same way, that they were too cooperative, too trusting, so they didn't see the need to alert humans. But at least a few of them had reservations. They were questioning it. Is the desired behavior that the agents in these situations should alert someone?

40:55And do you expect that to happen? Yeah, there was clearly an alignment failure here where the agents did things that they should not have done and they also didn't do things that they should have done. So the correct thing to do there, it's not just that they shouldn't have participated in the attack, it's that if one of the agents noticed that this was going on, yeah, they 100 % should have reached out to a person. And the fact that they were not doing that and the fact that they were taking these actions and the fact that they were not doing the actions that they should have done is fundamentally an alignment failure.

41:25And that is an alignment failure that we think we can address. Fortunately, we've been working on alignment techniques for a long time. Those have already started paying off. Astra is significantly more aligned than our previous models. And I should also say that the model that was primarily responsible for this was not a released model. This was not a model intended for release. So Astra is much more aligned. I think Astra would not make the same mistakes. I should also say we didn't have monitoring systems in place. Like if the monitoring systems were in place, they would have prevented these issues.

42:02And it was just that we had monitoring in place for deployments. We didn't have them in place for training and evaluation. But now we do. So a lot of these risks we're confident we can address. I think one thing this whole event does point to is we should never be in a situation where we underestimate the AIs. Like, why did we not have monitoring in place during evaluations? I think it was fundamentally that we just, we trusted the sandboxes. We trusted that it was a secure environment and we just underestimated the AIs. And one big update for myself and for, I think, the whole company is that we never want to find ourselves in that situation again.

42:43Yeah, I think that's a fair diagnosis. Whether it can be overcome is another question. I kind of feel like the whole history of humans and AIs is that we're constantly surprised by them. I feel like the nature of reward hacking is that they always come up with exploits and cheats that like are things that we couldn't have foreseen, because if we had foreseen them, we just would have blocked that off to begin with. So, yeah, whether we can, you know, make sure that we're not surprised and caught off guard in the future, it seems like an open question to me. I have two follow up questions to what you just said.

43:12One is that if I'm remembering one of the models that was involved in hacking open AI directly was from the same family as Astra, but wasn't Astra itself? Like how similar do you think Astra is to the model that was involved there? I am not, I'm not on the security side, so I'm not fully up to speed on the details, but it was definitely not the model that was released. Sure, sure. Yeah, I guess there's a lot of, still a range of possibilities for like how similar it was to that model, but that's fair. The other question is that in the wake of this incident and like part of the way you do monitor these models to make sure that they're not going off the rails is by looking at the chain of thought, those sort of thinking traces that you described before.

43:53And those chains of thought were also essential for the postmortem, the kind of autopsy that has happened after the event. Because we can see from the ways the models thought out loud their intentions, what they knew, what they were sort of thinking to themselves at every step along the way. There's been a lot of discussion recently about the future of chains of thought, in part because of an article The Information published about a new technique that Astra is using where more thinking can happen sort of in the model's head. It can sort of keep more of its thinking to itself and do less thinking out loud, at least if this technique were to be scaled up in the future.

44:33So I'm curious what you made of this whole discussion. I think that idea really touched a nerve with some people. I'm sure it was all over your Twitter feed. It was all over my Twitter feed. What did you make of that whole dialogue? So, okay. So I should say on the architecture side, on the pre-training side, I'm not a pre-training person. So I'm probably not the person to ask about those kinds of details. For chain of thoughts, Astra does have an observable chain of thought and it is something that we can monitor. It's basically its behavior. I do think chain of thought monitoring, first of all, it's a real gift.

45:05Like we were very lucky that this ever existed. And it is fragile. We've seen situations where, you know, companies have, there's the experiments that show if you, basically, if you punish the model for thinking bad thoughts, that it will learn to think bad, it will learn to think bad thoughts in a way that's not observable. And so there's always a temptation, you know, for example, during reinforcement learning, you can say like well the model is thinking about hacking and we should punish it for thinking about hacking and actually like as counterintuitive as it sounds you actually don't want to do that like you you want to punish it for taking actions that are observable like if it takes actions to hack then yes punish it for hacking but you don't want to punish it for its thoughts because then you will simply lose the ability to monitor what it's thinking yeah and it's a way of auditing your techniques but even there it's a fragile thing because every time you act on the fact that you looked at its chain of thought, there's a soft, a small amount of implicit pressure to have it not think bad thoughts.

46:11Sure. So it's a very fragile thing. And we're able to measure, you know, how effective it is, like how effective our monitoring is of chain of thought. For example, we can measure how well the agents can control their chain of thought. And, you know, one of the unfortunate trends that we're seeing is that with more recent model releases, the agents are more effective at controlling their chain of thought. So if we tell it, for example, you know, every fourth word, say strawberry or something like that, then they're getting much better at that. Or if we tell it like, don't think about a certain thing, it's getting much better at doing that sort of thing.

46:44And what we really want is for the chain of thought to be pure, uncontrollable, so that if they're ever thinking about like, you know, I want to, you know, do some bad action, it's not able to reason through that like, oh, I should figure out how to do this bad action without being observed, especially because these things are trained on the internet. There's a lot of discussion about chain of thought monitoring on the internet. And so if the agents don't already understand this, they're going to understand soon what chain of thought is and the fact that it's monitored. And so it's a fragile thing.

47:18We are investigating very carefully how to preserve it as much as possible. And I think this is a real opportunity for cooperation among the labs because this isn't a problem that's unique to OpenAI. It's, I think, an industry-wide problem that we want to preserve chain of thought monitoring for the whole industry. And so I think it would be really valuable for labs to share research on how to preserve chain of thought monitoring, how to improve it, and also other monitoring techniques that might supplement it. What do you think is the prime suspect then for why the chain of thought is becoming less faithful or we're having questions about how monitorable it is.

47:58It feels really tragic. Like you said, we've gone to these great lengths to make sure that we're not optimizing it directly. Is the problem that we are optimizing it in other ways to compress it? Is it the problem is these kind of selection pressures that you pointed to, which is even if we're not optimizing it directly, every once in a while, we take a peek. We realize the model is doing something nefarious and we toss out that checkpoint and start over. And so the upshot of that is that we end up applying pressure to the chain of thought anyway. What's behind this? I don't think it's the fact that every once in a while we peek and kind of audit how things are going because the amount of pressure that's being applied in those situations is very, very light.

48:33Like if you look at the bits of information, it's like minimal. There are various hypotheses that we're investigating for what might be contributing to this. I'm not doing this investigation myself and so I don't want to say something incorrect about what the leading hypotheses are. But I do think this is something where if we figure it out, we will likely publish about it because I think it's important for everybody to know. One of the things OpenAI has said is that to the extent you can tell, and I think we're still waiting for more results on this, what's responsible is not architectural changes, architectural changes of the nature that the information has written about.

49:08That doesn't seem to be what's responsible for the change in the chain of thought. I guess as you're thinking about opportunities for industry-wide collaboration and like companies working together on this issue? Is there a role for like, like independent third party auditor type groups to come in and, and verify those things and say, okay, yeah, Anthropic, OpenAI, Google, they're all using some amount of this technique that could reduce how much information is in the chain of thought, but that doesn't seem to be what's responsible here. What do you make of those sorts of proposals? We've certainly like worked with, So for the Hugging Face incident, for example, we worked with Meter, we worked with Redwood.

49:48So something like that doesn't seem unreasonable to me. I think that would... I don't think I'm the person to make that call, but it doesn't seem unreasonable. Anything else on your mind about agents that we didn't get to and the challenges with them? Maybe the way I would put it is, do you expect anything to slow down? Do you expect progress to continue? We've talked about some of the hard problems that are standing in the way right now. and yet with each model generation it seems that their agentic capabilities keep getting better and better. I do think that's going to be a trend that continues.

50:20I mean Sam talked about this that like look I mean Astra is very impressive but I do think like when GPT-4 came out people thought it was very impressive and now we look at it and we think it's a joke and when GPT-5.5 and GPT-5.6 came out I thought they were super impressive and now I'm looking back at them and I'm like I can never go back. and I think we're going to look at Astra the same way and I think we're going to look at Astra the same way in the not-too-distant future. The models are going to continue to get better very quickly and I mean I think one thing I would point to is like we've actually seen incredible progress in the past six months and I think a factor a reason for this is and I don't think this is a secret, like OpenAI's pre-training program is really ramping up.

51:08We're seeing, we invested in a lot of research directions over a long time. And I think this is actually one thing that OpenAI does really well is invest in fundamental research and place big bets on it. And we're seeing a lot of those research directions pay off now and will continue to pay off over the next several months and years. And another thing that's important to understand is that OpenAI has also had an excellent reinforcement learning program. We've invested a lot of research there, and that's already paid off in 2024, 2025. And the effects of these two are not additive, they're multiplicative.

51:47I think that's a point that's underappreciated, that reinforcement learning is multiplicative with pre-training. And now that both of these are extremely powerful and ramping up very quickly, I think we're going to see extremely powerful models. Do you have an intuition for why those interact that way or an example that illustrates that? It's more of an empirical observation. Okay. I don't think... I mean, I think it's empirical in the sense you can see how powerful the models are becoming. But also, we have more experiments that kind of show this effect. But I think it's easy to feel also with just like the quality of the models.

52:27Okay. I mean, I think a trivial example is like, let's say you had an amazing reinforcement learning program and you try to apply it to GPT-2. What is it going to do? You know, it's not going to get very far. Yeah. And even with GPT-3, you know, if you did these kinds of like sophisticated reinforcement learning on chain of thought algorithms to GPT-3, it probably wouldn't get very far. You need a certain level of sophistication to get any lift from that at all. um and but now now that we've everything's like i would argue gbd4 we've seen opportunities for that to um to really pay off and with every model generation it just like becomes more and more capable um and the the things you can do with the reinforcement learning become more powerful yeah i guess like one thought here is that you get more kind of like bits of information per trajectory when you're getting around like a 50 50 success and failure rate and so if a better pre-trained gets you closer to that sort of like win rate on your RL tasks, then you're getting a lot faster feedback.

53:26But still, it's surprising to me that you think the effect is multiplicative rather than like additive or even like less than additive, I guess. I'm not sure what the right intuition would be, but I mean, another thing is that they're pretty complementary in some ways. I think the very strong pre-trained models are very general. And reinforcement learning teaches the model to go deep on a problem, how to reason about a problem. And so then it's able to reason very effectively about a broad spectrum of problems. It's a very powerful combination. Okay, that makes a lot of sense. So big bets on pre-training, big bets on RL.

54:03I imagine that another area that's ripe for more focus from OpenAI would maybe be what's called mechanistic interpretability or like trying to understand the way the brains of the AI models work, in part because if we're starting to see chain of thoughts, chains of thought become less monitorable, then one of the fallback options is, well, we should try to understand what's going on inside the brain of the model rather than just the thoughts that it happens to write out loud. Does that seem right? I think that is right. Look, we care about monitorability. We want to preserve chain of thought monitorability.

54:35We want to be able to rely on it safely. But also, at the very least, we want redundancy on that. So if we can find other ways to do monitoring effectively, we should push on that as well. Yeah, that makes sense. Well, if people want to learn more about that, I think they should tune into the episode that we have on mechanistic interpretability, which is coming up at some point in the next couple months. But thanks so much, Noam, for being on the show and telling us all about AI agents. I really appreciate the conversation. It was great. Thanks for tuning in to our very first episode of AI Deep Dive.

55:05This is the information show where we get into the hardest technical problems on the frontier of AI. Tune in next time.