In short
The Information TV episode covers (1) OpenAI’s Cerebras deal that could let OpenAI capture about $1.3B+ in Cerebras equity via $0.0001 “penny” stock warrants tied to a ~$20B multi-year compute commitment; (2) OpenAI’s renegotiated revenue-sharing with Microsoft, including a $38B cap through 2030 and removal of an AGI clause; (3) testimony in the Elon Musk v. OpenAI/Microsoft trial; (4) Replit’s AI coding model benchmarking and security; (5) Thinking Machines Lab’s “interaction models” research preview.
Guests and backgrounds
Corey Weinberg (Deputy Bureau Chief of Finance, The Information). CJ Gustafsson (Mostly Metrics newsletter author). Aaron Holmes (Microsoft reporter, The Information). RocketDrew (covers Musk-OpenAI trial). Michele Katasta (Replit president and head of AI). Stephanie Palazzolo (AI reporter, The Information).
Key claims/examples
OpenAI’s warrants could scale to ~10–11% ownership; Cerebras IPO valuation comps shift from ~60x current revenue to ~10–13x forward revenue due to OpenAI-backed backlog. Microsoft gained at least ~$30B (2023–2025) from Azure server rentals, Copilot, and model sales; OpenAI shared ~20% of revenue with Microsoft but now capped at $38B total. Nadella clarified “below/above/around” compute vs apps/APIs; Ilya said OpenAI’s mission is larger than structure and expressed reservations about Musk controlling a for-profit subsidiary. Replit launched BiBench (bibench.ai) to end-to-end test coding agents; it uses security scanners and dependency pinning. Thinking Machines’ interaction models take audio/video/text continuously for real-time interruption, compared to OpenAI’s bidirectional voice efforts.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Chapters
Tap a time to open that second in VOCerebrus IPO Insights
0:46 to 4:02
Discussion on OpenAI's stake in Cerebrus ahead of its IPO.
“I'm going to chat with Replit's head of AI about how his company is benchmarking coding models, and we will wrap the show with a look at a research preview of Thinking Machines Lab new model.”
Unpacking the Deal Mechanics
4:03 to 7:04
Exploration of the details and implications of OpenAI's deal with Cerebrus.
“maybe this is a stupid question, but why wasn't it enough just to say, I mean, they're going to provide the chips at some point, right?”
Comparing OpenAI's Equity Strategies
7:05 to 9:01
Comparison of OpenAI's agreements with different companies and their impacts.
“That was another comparison point you used in the story.”
Market Valuation and IPO Expectations
9:02 to 11:50
Discussion on Cerebrus's market valuation and its comparison to industry standards.
“and why I think the Cerebris deal is even more aggressive in favor of OpenAI is that OpenAI's equity agreement with CoreWeave doesn't scale over time.”
New Developments in OpenAI-Microsoft Agreement
11:51 to 13:26
Insights into the changes in OpenAI's revenue-sharing deal with Microsoft.
“What are some of the factors we should be watching for?”
OpenAI-Microsoft Revenue Sharing Agreement
14:02 to 15:00
Learn about the new revenue sharing cap set for OpenAI in its agreement with Microsoft.
“And what we found out just this week is that as part of their new deal, OpenAI has a cap on the amount of revenue that it has to share with Microsoft.”
Winners and Losers of the Deal
15:00 to 16:20
Discussion on who benefits from the openAI-Microsoft agreement amidst the changes.
“So$38 billion is sort of the maximum amount of revenue that OpenAI will share with Microsoft under this arrangement?”
Microsoft's Focus on AI Profitability
16:20 to 18:22
Explore how Microsoft shifts its focus from AGI to profitability in AI.
“or is it really just like the CFOs who have a little bit of certainty as far as what they're getting?”
Microsoft's Revenue Gains from OpenAI
18:22 to 20:17
Insights into how much revenue Microsoft gains through OpenAI-related activities.
“Tell me a little bit about what you found there.”
Understanding Microsoft's AI Revenue Streams
20:17 to 22:41
Analysis of the various revenue streams Microsoft has from OpenAI's services.
“We know that OpenAI spent the vast majority of the roughly$22 billion through 2023 to 2025 on Azure.”
Show all 17 chapters
Evaluating the Cost of Microsoft's Investment
22:41 to 24:22
A look at what Microsoft is giving up with its investment in OpenAI.
“Now, I do want to be fair here because we're talking, I mean, you have a lot of reporting on how Microsoft is benefiting from this arrangement.”
Testimonies in Musk-OpenAI Trial
24:22 to 28:05
Details on the testimonies from Satya Nadella and Ilya Sutskover in the trial.
“I mean, again, this is like a product versus distribution question, right?”
Musk v. Altman: Key Testimonies
28:05 to 29:20
Exploring the implications of recent testimonies in the Musk-OpenAI lawsuit.
“He ended up saying something that probably advances OpenAI's side in the lawsuit.”
AI Model Benchmarking with Michele Katasta
29:42 to 39:03
Michele Katasta discusses AI benchmarking and model selection at Replit.
“I want to ask you a very basic question.”
Consolidation in AI Coding Landscape
39:03 to 39:21
Discussion on the competitive landscape and consolidation in AI coding.
“But I think we're going to be seeing more and more engineering effort happening on top of models rather than just focusing on creating new models.”
Thinking Machines' Interaction Models
39:21 to 42:00
Stephanie Palazzolo explains new interaction models from Thinking Machines.
“Thinking Machines Lab released a research preview of its new AI interaction models.”
AI Models and Natural Conversation
42:00 to 44:22
Explore how AI models are evolving to communicate more naturally like humans.
“or to give it new information, the model often gets confused and will just completely stop talking, might ask you to repeat yourself.”
Transcript
Automatic transcript. May contain errors.0:13Welcome everyone to the information's TI TV. My name is Akash Pasricha. It is Tuesday, May 12th. First up today, chipmaker Cerebrus is set to make its public debut this Thursday. Our Deputy Bureau Chief of Finance published exclusive reporting today about how much OpenAI stands to gain from the IPO. We'll talk about that shortly. We'll then turn to more exclusive reporting around OpenAI's revenue sharing agreement with Microsoft. We'll also get the latest on the Musk OpenAI trial from RocketDrew. We've also got a conversation for you in partnership with Gemini. I'm going to chat with Replit's head of AI about how his company is benchmarking coding models, and we will wrap the show with a look at a research preview of Thinking Machines Lab new model.
1:03It's going to be a fun show, so let's get right on into it. As Cerebris marches towards its IPO this week, our newsroom has been looking at who stands to gain most in the offering based on how much equity they already have. Our Deputy Bureau Chief of Finance, Corey Weinberg, has a story out today explaining how OpenAI landed itself in that position. I want to bring him on to talk about it, and I also want to bring on CJ Gustafsson, author of newsletter Mostly Metrics. Welcome to you both. It is great to have you here. Corey, I want to start with you. So how much does OpenAI stand to gain in the Cerebris IPO?
1:41open ai has a surprisingly large stake that they have amassed in a very short amount of time within the next month or so probably after cerebrus's ipo open ai should have a stake worth about 1.3 billion and it could scale by billions more over time this this was a deal that they reached with cerebris back in december uh where they took essentially these incredibly cheap uh stock warrants at a fraction of a penny uh just in in exchange for promising to become a huge cerebris customer over time so can you explain the mechanics of that a little more in in detail here and how how did it amass this stake and what did it give up Yeah, it's essentially a sign of how much leverage OpenAI has gained among some of its suppliers.
2:47Essentially, it and Cerebris came together and came up with a somewhat creative structure so that OpenAI could essentially guarantee more compute capacity over the next few years with a company not named NVIDIA, which OpenAI is obviously incredibly reliant on. Cerebris is trying to challenge NVIDIA as the role of inference compute provider. And in exchange for OpenAI making a pretty massive $20 billion multi-year commitment, OpenAI got a pretty huge sort of sweetener on the deal, which is a stock warrant grant at$0.0001 that would allow them to amass 33 million shares over the next several years.
3:42It will have invested about nearly a third of that probably within the next month just by lending Cerebris about a billion dollars for working capital, as well as Cerebris surpassing a$40 billion market cap. So let me understand this. So the working capital loan, the condition around the market cap, maybe this is a stupid question, but why wasn't it enough just to say, I mean, they're going to provide the chips at some point, right? So, I mean, what? They needed, OpenAI needed equity as sort of a, you know, like a backup plan if Cerebris couldn't deliver the chips? Or what's the idea here? The simple idea, Kosh, I think, is that Cerebrus needed OpenAI more than OpenAI needed Cerebrus.
4:34I called it a – the warrants are certainly a sweetener, and it is like a pretty sweet deal. And, you know, these aren't completely unheard of in the commercial growth equity world. penny warrants do exist, but the size of this deal, OpenAI potentially getting 10%, 11 % of Cerebris is pretty unusual. And I think it speaks to the leverage that OpenAI has as a buyer. Okay. So CJ, I want to come to you. Help us put this into context. Your newsletter dives into all the many things that CFOs need to think about in the current moment. this is kind of a deal that i haven't heard of before i don't know cory's saying many maybe penny warrants are a thing is how unconventional is this this is pretty unconventional aws also is warrants but in order to exercise those it's a more commercial agreement where they'd actually have to pay over 200 million dollars so that part makes sense and that was a big deal that they signed in March of this year, which I think is another reason why the stock price could be higher once the IPO.
5:45But this is extremely strange. To Corey's point, he was listing out the decimal points that it goes to. You have to like stare at it and count how far to the right it goes. And I did the math. It's essentially, it could potentially be worth$5.1 billion in equity if everything goes right across the three tranches. And it would only cost him about$350, which because about the price it costs to take my family to dinner. In the story, CJ, you offered this quote when talking about the dynamics of the deal. If I'm in Cerebris' shoes, maybe it's a deal I had to take to a certain extent to get the scale that I want to be.
6:27Can you unpack that a bit? Yeah, and I want to give them the benefit of the doubt because amazing companies that are infrastructure and capital intensive don't come about without some sort of propping up from another force. And it's not just OpenAI that's helping them with this working capital loan of a billion dollars. They have two major customers who are also investors, who also prepay for things to be built in Saudi Arabia. And those deals, you kind of squint and look at them as well. And you're like, well, how did this come to be? Is this exactly by the books? If you have two customers and make up 85 % of your revenue, but you have this cold start problem when you're creating something from scratch and these chips i mean they take a lot of investment the company's been around since 2016 and uh you know the founders have been working hard at it and been diluted to a certain extent because of what it takes to make but i mean like in in order to build something great sometimes you have to take a deal that that looks a little funky cory how does this contrast with uh you know the the deals that OpenAI has done with CoreWeave, for example.
7:34That was another comparison point you used in the story. Yeah, that's how I sort of landed on this topic and found it interesting, was I covered the CoreWeave IPO back in March of 2025. And I was sort of sifting through the Cerebris IPO filing in recent weeks and just sort of reading all the details of their OpenAI deal and having flashbacks. Because this was a big part of the CoreWeave IPO that I wanted to understand better as well. Essentially, what happened about 14 months ago when Corwee went public was they were not getting amazing feedback from Wall Street ahead of their IPO. And there had been a lot of reports around the time that their relationship with Microsoft was not in the best shape.
8:25What really helped turn around sentiment was that they had landed this large agreement with OpenAI to buy their cloud computing services. OpenAI agreed to buy about, I think,$12 billion worth of those services over the next five years. And in exchange, OpenAI got a big slug of shares in a private placement right after the IPO. So these weren't penny warrants, exactly. these were for free. OpenAI just got these shares. What was different with the CoreWeave IPO and why I think the Cerebris deal is even more aggressive in favor of OpenAI is that OpenAI's equity agreement with CoreWeave doesn't scale over time.
9:14They got these shares in the company and they don't have sort of an allotment to get more as they buy more. But Cerebris, this deal scales to potentially 10 % or 11 % of the company. And so you see, I thought it was very interesting how you saw over time OpenAI doing a structure like this and then realizing what we're buying has a ton of value and we should be getting more upside. And so you saw that with a deal they did later in the fall with AMD, and now it's Cerebris, where these deals can get really supersized. CJ, I'm sort of thinking about where Cerebris is at in its company lifecycle. This was a deal that, as Corey pointed out, I mean, they had to get business from OpenAI, and this is what they had to give up.
10:08If you look at the current state of the business, I mean is this a very was this a sort of an early stage type of deal to do or or and I'm putting this out there because I think we know better than to than to predict the end of circular deals and circular financing. Could you see more of these types of deals coming you know for Srebres even after they go public. I mean talk about that. I don't think this is the end of the circular deals. And by the way, Corey or someone from the information should do a story on if OpenAI is really just a venture company that happens to make large language models because they've got to have the best venture portfolio out of anybody out there.
10:52It's absolutely incredible. And at the best cost, too, right? Their cost basis must be off the charts. But I don't think this is what a press release, really. Yeah, right? So they're doing something right there. I don't think this is the end of these types of arrangements, because if you look at the ecosystem, there are players that benefit from like the slack capacity that others have. Like you could even extrapolate this out to the SpaceX IPO and being like, well, why are there so many different companies that are bundled in this? It's like, well, XAI might play well in terms of its data center capacity for Kurser to come on with their call option.
11:28So I don't think this is the end of it. What I am curious to see, though, is if they can get a premium on a backlog of revenue of 24 and change billion dollars where the bulk of it is just open AI. Right. And it's on their nascent cloud business, which is the smaller component of their revenue today. So I don't know. I don't know how that plays out. Right. Right. And I mean, just talk a little bit more about what you're watching for in terms of the Cerebrus IPO and the current state of the company. What are some of the factors we should be watching for? Yeah, so every week I track the valuation multiples of 142 companies in my newsletter.
12:07And if you were to take their 2025 revenue right now and say, well, what would the revenue multiple be on$33 billion, which is the midpoint of what they priced at, that would be like a 60x current revenue. But because of this backlog, which goes live, assuming that the deal with OpenAI is on the up and up and continues to have enough capacity, it puts them at about$2.5 to$3 billion in revenue. So it becomes a more palatable 10 to 13x forward revenue. And for context, the median tech company right now is trading at 3x. The top 10 are trading at 14x. And then you have Palantir that's around 50x, which is number one.
12:45And then you go down to Cloudflare, which is 30x. So if you were to say, what's the universe of comps that it's most relevant to? They want to be at a premium to core weave which is i think interesting core weave is trading about 9x and change forward revenue they want to be somewhere in the 10 to 13x which would also put them ahead of amd but below nvidia ahead of amd but below nvidia okay well uh that is a tall ambition and i think we're all excited to see how it trades come thursday i want to thank you both for coming on that is cory weinberg our deputy bureau chief of finance and cj gustafson from the mostly metrics newsletter here on ti tv The latest iteration of OpenAI's revenue-sharing deal with Microsoft marked a significant shift in the partnership between the two players.
13:33My colleagues Shreem Uphidi and Aaron Holmes have some inside reporting on who the winners and losers are in this latest arrangement. I want to bring on Aaron Holmes to talk about it. Aaron, welcome back to the show. It's great to have you here. Happy to be here, as always. What did you find with Sri with respect to OpenAI and Microsoft's new revenue sharing agreement? So we already knew that OpenAI and Microsoft renegotiated their partnership in April. And what we found out just this week is that as part of their new deal, OpenAI has a cap on the amount of revenue that it has to share with Microsoft.
14:12And specifically, that cap is$38 billion. So we knew that OpenAI was sharing 20 % of its revenue with Microsoft. And according to OpenAI's internal revenue projections, it could have shared over$100 billion of revenue if it hit all of its revenue goals between now and 2030. Now it's only having to share$38 billion. So that's pretty significant savings for OpenAI. Of course, Microsoft is getting some things in exchange. For example, now all of the payments have to happen between now and 2030. Previously, they were going to be deferred over a longer time period. And they did away with the AGI clause, which could have taken away Microsoft's exclusive license to the technology when OpenAI hit AGI, and that's no longer a part of the agreement.
15:00So$38 billion is sort of the maximum amount of revenue that OpenAI will share with Microsoft under this arrangement? Yes, in addition to what it's already shared, which is close to$3 billion so far, that's going to be the total amount that it will share between now and 2030. Right. So basically, they basically replaced the AGI cap with sort of a hard number in saying$38 billion, or they've also put a time limit on it now, which is 2030, I believe you said. Yep, that's right. Yeah, now OpenAI is basically on the hook to keep paying 20 % of its revenue on schedule between now and 2030, instead of being able to sort of spread out those payments over a long time period, which it was allowed to previously.
15:54So now let's go back to what you were talking about earlier, which is the winners and losers of this. I mean, let's talk about this. Microsoft, it, I guess, depending on when you thought ATI was going to be reached, which was the whole, you know, gray distinction here in the first place. It's either getting more or less than it would have. I mean, I guess really, are there winners and losers here? or is it really just like the CFOs who have a little bit of certainty as far as what they're getting? You know, I think it's safe to assume that Microsoft wouldn't have agreed to this deal unless it felt like it was a safer bet.
16:34I think that truly, you know, what Microsoft gets is just this deal of certainty that, you know, specifically, I think having the IP license, which goes now through 2032 and is not tied to AGI, is potentially, you know, even more valuable to Microsoft than getting a cut of OpenAI's revenue, especially because, you know, being able to keep selling OpenAI's models to its own customers could be a much larger source of revenue, including through, you know, its co-pilot software and also just selling the models themselves than simply getting these cash payments that were going to end at a certain point anyway.
17:10It's also kind of funny to me, Aaron, that, you know, maybe a year ago, the conversation around AGI, when you will reach AGI, what the definition of AGI is. I mean, this was like the big question that everyone was concerning themselves with. It feels like in the past year, I mean, that's certainly fallen off. And I think it's fallen off in favor of people saying, it doesn't matter because we have to make money. You know, whether it's AGI or not, we have to sell what we have. And so it very much feels like people have basically acknowledged that this was a useless goal in the first place. Yeah, and it's interesting because Satya Nadella has actually been banging that drum kind of in public for years saying that he, you know, isn't really interested in defining when AI is as intelligent as a human or as capable as a human worker and that he was more interested in, you know, when AI will generate profits and when it will contribute to GDP.
18:06And I think that this renegotiated deal is maybe kind of a reflection of that philosophy, specifically just that, you know, Microsoft is a business. It wants to be able to make money from this investment. And that's now what they're more focused on. Okay, so let's pivot now to another story that you published about how much Microsoft stands to gain from OpenAI in terms of the cloud business. Tell me a little bit about what you found there. Yeah, so we learned some new details about how much OpenAI has been paying Microsoft to rent cloud servers and through that revenue share agreement that we mentioned earlier.
18:45And then, you know, took those numbers as well as some informed estimates about how much revenue Microsoft is making from its own sales of Copilot and from its sales of OpenAI models. And the high-level idea there is that between 2023 and 2025, Microsoft has made at least around$30 billion in new revenue from all of those OpenAI-related revenue streams. And, you know, that's notable because obviously Microsoft invested a total of just over$13 billion into OpenAI. And in just the last two-ish years, they've gotten more than double that investment in the form of new revenue from selling these products as well as from OpenAI as a customer.
19:29And so just to make clear for folks here, what we're talking about here, we're not talking about the gain on the equity stake that Microsoft has earned from that$13 billion investment. We're just talking about new revenue from OpenAI that has come back to them, which is totally separate of how much money their stake would have appreciated to me. Right. And yeah, our understanding is that the stake itself, as of, you know, OpenAI's last funding round that was announced in April is now worth over$200 billion. So it's safe to say that that has also appreciated pretty well. Of course, that's just on paper, whereas the more than$30 billion that we tallied is actual revenue flowing, you know, into Microsoft's coffers from customers.
20:17right and so so let's now let's talk about about because there is certainly the revenue from open ai but then there is also the added benefit that microsoft gets to sell open ai's models to its customers and incur more revenue there do we have any reporting on how much revenue it's been able to incur on on that uh revenue stream yeah so the the bulk of the open ai related revenue we can call it, came directly from OpenAI paying for Azure servers. We know that OpenAI spent the vast majority of the roughly$22 billion through 2023 to 2025 on Azure. And so, you know, Azure has gotten effectively$20 billion from OpenAI just on server rentals in that period.
21:04And then on top of that, Microsoft has the rights to sell OpenAI's models on Azure to its own customers. And those sales have totaled at least$2 billion from 2023 to 2025, you know, plus the$3 billion that they have gotten from the cut of OpenAI's revenue, and then, you know, several more billion from sales of Copilot. That brings us to over$30 billion in those two years. So how$30 billion, I mean, put that into context of Microsoft's business as a whole, how significant is that? You know, it's not enormous for Microsoft's business. I mean, the company does several hundred billion dollars of revenue annually and is growing quickly.
21:50But it does, you know, kind of speak to how important OpenAI has been specifically for powering its AI revenue. And, you know, something that Microsoft disclosed in its earnings call last month was that they have basically in the neighborhood of roughly, you know,$37 billion in what they're calling AI revenue, which is just all of the money that they're making from renting GPU servers and from selling Copilot and other services that they sell directly to big labs like OpenAI and Anthropic. And, you know, this analysis that we've done shows that the majority of that has come from specifically, you know, these open AI tied revenue streams.
22:31And it essentially kind of reflects the benefits that Microsoft has gotten from making this somewhat risky bet on investing$13 billion into open AI a few years ago. Now, I do want to be fair here because we're talking, I mean, you have a lot of reporting on how Microsoft is benefiting from this arrangement. But are there ways in which Microsoft is sort of giving something up other than the$13 billion initial investment? I mean, it's not like the people at OpenAI are just sitting around, you know, giving up everything. What is Microsoft giving up here? What does OpenAI gain through all this? I mean, I think that, you know, arguably, if Microsoft hadn't invested this much cash and also built supercomputers for OpenAI, I don't think it would have been necessarily possible for OpenAI's business to grow as quickly as it has.
23:22At the same time, you know, we have seen that OpenAI is, you know, accelerating its own revenue a bit faster than Microsoft's AI model sales. So specifically, we learned that, you know, the Azure OpenAI service, which is Microsoft selling OpenAI as models, that revenue was around$1.5 billion last year, whereas OpenAI's own sales of its models directly to customers were over$2 billion. So, you know, there is seemingly more interest from customers to buy models directly from OpenAI than to buy OpenAI's models through Microsoft. But at the same time, this has obviously been very beneficial to both companies.
24:04Well, I mean, that would make sense too, right? And I guess that is the threat, I guess, to Microsoft is if OpenAI can basically find a way to continue that traction and make its product more compelling, having a direct relationship with the customers. I mean, that could then in turn affect that part of that OpenAI related revenue, which is the reselling of the models. I mean, again, this is like a product versus distribution question, right? Is it distribution that wins? Is it product that wins? So that is certainly an interesting question. A lot of great reporting in there in both those stories.
24:46I want to thank you for coming on, and we will certainly have you back with Shreema Petey, who was a co-author on both of those pieces. That is Aaron Holmes, our Microsoft reporter, here at The Information. The information's Rocket Drew is covering the Elon Musk OpenAI trial from the courtroom all week long. Here is his latest update. Satya Nadella and Ilya Sutskover testified in Elon Musk's lawsuit against OpenAI. Both of them have been highly anticipated witnesses in the trial because of their unique perspectives on OpenAI. Satya Nadella is the CEO of Microsoft. Microsoft is a defendant in the case.
25:28Musk alleges that Microsoft aided and abetted breaches of charitable trust that OpenAI committed, meaning when OpenAI allegedly acted in ways that were contrary to its nonprofit mission, Musk alleges that Microsoft assisted in that. So in part, that's by investing in OpenAI and taking the rights to some of OpenAI's IP and the rights to commercialize its technology. It's also by providing input about who OpenAI should elect to its board of directors. Satya exchanged some texts with Sam Altman around the time that Sam Altman was fired from his position as CEO of OpenAI. Musk alleges that this is sort of undue influence that Microsoft has over what should be an independent charity The most important takeaway from Satya's testimony was that he actually had the opportunity to explain some comments that he gave to the press around the time that Sam was fired Satya had said, we're below them, we're above them, around them which Musk's side has been presenting as a statement of control that Microsoft sort of holds all the cards in the relationship with OpenAI.
26:39Satya explained that what he meant by those comments was when he says we're below them, that means Microsoft provides the underlying compute infrastructure that OpenAI builds on. When he says we're above them, he means they develop applications on top of OpenAI's technology. And when he says they're around them, he means they provide them APIs and other developer tools to do their work. So that may have gotten Satya out of some hot water here, and that could prove important for Microsoft's defense in the case. Ilya was kind of a wild card, though, because he has affiliations with both sides. On the one hand, you might expect him to sympathize a little bit with Musk's case, because Ilya played a crucial role in the decisions that led to Sam Altman getting fired from the board.
27:22Ilya was a co-founder of OpenAI and served for a time as its chief scientist, but he eventually left OpenAI after flip-flopping and deciding to bring, to advocate for Sam Altman to return to the company, and then Ilya eventually left. So he has some affiliation with Musk's side. On the other hand, he was a co-founder, he was a board member, OpenAI is paying his legal fees, it's natural to expect that he has some allegiance there as well. In practice, I think Ilya basically came out as independent from both of the parties. He sort of has his own point of view on things. He certainly had some concerns with Sam leading up to the firing, but ultimately what he cared about was OpenAI, and at the end of the day, he wanted to make sure that OpenAI continued to exist and continued to pursue its mission.
28:11He ended up saying something that probably advances OpenAI's side in the lawsuit. He said that OpenAI's mission is larger than any structure. And he talked about how he was supportive of some different structures that were being considered back in the day when different structures for OpenAI were under consideration. But he said that he had reservations about Elon Musk having full control or a majority of the equity in an OpenAI for-profit subsidiary that he had concerns that that wouldn't be fair, given the amount of time that Musk would be able to dedicate to OpenAI among his other pursuits.
28:46So that was the testimony most recently in Musk v. Altman. Up next, we're expecting to hear some testimony from Sam Altman himself, who has certainly been a highly anticipated witness. A lot of the events concerning Sam have become the crux of this case, including the events surrounding November 2023 when he was fired. So it will be very interesting to see how he explains and how he accounts for those events when he testifies at the trial. Our next segment is supported by Google Gemini. As AI models become more powerful, companies are increasingly focused on how to measure that performance. Benchmarking is certainly an important part of that story.
29:33Joining me now to discuss how Replit thinks about that is Michele Katasta, Replit's president and head of AI. Michele, it's great to have you here. I want to ask you a very basic question. How do you decide which models to offer your customers? AI coding models, I mean, there's so many of them. How do you even think about which ones are competitive and how that changes over time? You're right. The choice is very wide. And I would say over time, it's more and more of a challenge to pick the best model for whatever our agent wants to do. We spent quite a lot of research effort, especially in the last six months, and we decided to take the burden on ourselves and create a new benchmark end-to-end for pipe coding.
30:16We literally launched it last week. It's called BiBench. You can find it at bibench.ai. And the key insight is we created something that allows you to, first of all, instruct the agent to build an application zero to one, as our users do on our platform on a daily basis. And then at the very end of the process, we have another AI instructed to test step by step if the application works correctly. It kind of acts exactly as a human will do. You know, it opens a browser, it interacts around with different forms and buttons, and then it follows a rubric of how that application should actually behave.
30:52So we automated completely the process end to end. And then what we can do at this point is we can literally swap different model types, different combinations on how these models interact with each other and figure out which one of these configurations scores the best. In all honesty, I also wish it was this easy, while in reality, we have several different targets that we try to accomplish when we run evaluations. And as you can imagine, it's not only important to have high performance, but also speed of execution matters for users, as well as how affordable we can make our product. So we always have these like three competing dimensions at play.
Read the full transcript
31:31Right. And, you know, we need to decide what is the product philosophy that we want to follow. So on the affordability piece, I mean, this is interesting. You've built this benchmark. You know, is it always the case that you basically go to your customers and say, hey, whichever one of the top three models, these are the ones you should be using? Or is this more of a consulting conversation where it says, look, this is the best model right now. This is the frontier model. but you know maybe uh because of cost efficiency maybe because of what you need it makes more sense to take one that is lower down i mean take me inside those conversations so since day one when we launched rapid agent b1 back in september 2024 we made a product choice of not exposing a model selector which is really something that differentiated us from many other products on the market.
32:23I don't want to expose the level of complexity in front of our users because in all honesty, picking a model based on the model name or the provider is not trivial for a non-technical user. They are not supposed to know what is the state of the art, which frontier model is better what. So what we have in product today on Rapid is three different agent modes. We call them respectively light, economy, and power. Each one of them comes with a different price point, with a different capability in terms of how far you can push your application in that mode. And we craft under the hood exactly what is the right combination of models and how they should be interacting with each other.
33:03And we constantly evolve that configuration based on the batches that we run, our understanding of how models are evolving, as well as a lot of A-B testing in our backend. So our users are oftentimes put in front of slightly different configurations and we can understand better, you know, what is the right combination of models to utilize. So I am curious about how you think about your relationship with the labs because I'm sure you're talking to them all the time as you use all their models. You're also putting together this benchmark and you have to maintain a degree of objectivity. So, I mean, you know, just tell me a little bit about the conversations that you're having with them and how you manage that.
33:46So since we created ByBench, it's one of the main tools that we use to drive our partnership with Frontier Labs. For example, we give access to early models, maybe one or two months in advance, and one of the first things we try is to see how it scores on ByBench, which tells us both how powerful the model is, as well as how affordable we're going to be able to make our product with that. For example, in the case of Google Gemini, it's a model that is very competent when it comes to UI design. And that's one of the dimensions where we score every single application that we build on Rapid. So since we started to test Gemini Pro 3.0 and 3.1, we realized that we had a new tool in our hands that we could have used to really push forward the UI capabilities of Rapid.
34:37And it keeps us, I would say, objective because rather than doing just vibe checks, which is what we were used to do back in the days, now we can really assign an objective score to every single thing that Replit agent does. I want to ask you another question. So security has been an issue that has been top of mind in the AI landscape. We've seen vibe coding take off. Replit is in the business of AI-enabled coding. How do you ensure that security is prioritized? And I ask because it seems like as more and more code is generated through AI applications, we are seeing more security risks and security incidents.
35:22I'm fortunate enough to agree with you. We are undergoing the largest cybersecurity attack ever experienced. And not us as a company, but the entire space. And this is due to the fact that frontier models have also become exceptionally good at funding issues in code bases. So in light of that, at RapidWeb, we've been working productively towards this problem years in advance. Even before we launched Agent, whenever we built our infrastructure end-to-end, we always kept security as one of our pillars when it comes to designing our product. We have several different security launches that we focus on in the last few months, especially for our non-technical users.
36:03For example, we have an AI security scanner that you can run at several points in times during the app development process, especially before you decide to publish it either internally at your company or even potentially more dangerously in front of everyone on the internet. And these scanners are, they basically have the same kind of technology that attackers are using today to find vulnerabilities in their code base. Another thing that I'm very excited about that we launched a couple of weeks ago, we have the security center on Rabbit, which allows you to see all the different dependencies that your application uses and warns you in case any of them has been found to be vulnerable.
36:43You're probably hearing all the supply chain attacks happening in our space. The last big one has been literally just yesterday, and none of our users have been affected by the supply chain attack because in our system, we make sure that we pin certain dependencies. We always take care of making sure that everything is secure by default. And in the unfortunate case, that doesn't happen. We can bump up every single lab re-installation across our entire user base very quickly. So a lot of engineering investment that we've done in the past is paying off more and more now, where cybersecurity is probably going to be our top one priority at Rapid for a long time.
37:23Let me ask you one more question before I let you go. You know, AI coding is a very competitive landscape, and we see some consolidation in the sector. I'm, of course, pointing to the XAI cursor deal. Again, still, we have to see how it all plays out. But I think consolidation is inevitable, and the big players will continue to get bigger. How do you think about your competitive advantage against some of these big players? We think about it in a different way compared, say, to Cursor. Right now, we don't feel a need to create our own frontier model that drives the entire product. Rather, the competitive advantage is when you get closer and closer to a special enterprise customer, that's when you're exposed to the real nature, the complexity of the problem.
38:13Now, we're not just creating a coding agent that allows them to create a code base. Rather, we're creating a product for non-technical users that allows them to get work done. And that means that we have to create connectors, we have to have a certain data governance in place, we have to allow them to build different types of software artifacts, not just web applications. So the closer we get to their needs, the more we create a user experience that is very advanced, very custom-tailored for enterprise customers, and that is what has been differentiating us since day one. I don't rule out the option for us to create models, especially for certain capabilities of our agent.
38:52But we have a lot of work to be done just by caring about the application layer. And I think a lot of value will accrue more and more in the space where we are today. So yes, consolidation is going to happen. But I think we're going to be seeing more and more engineering effort happening on top of models rather than just focusing on creating new models. Great. Well, Michele, I want to thank you for coming on. That is Mikayla Katasta, President and Head of AI at Replit here on TI TV. Thinking Machines Lab released a research preview of its new AI interaction models. Joining me now to discuss the news is Stephanie Palazzolo, one of our AI reporters here at The Information.
39:32Stephanie, we've got this interesting news here from Mira Mirati and Company. What do we need to know about this new model? Yeah. So as you mentioned, this is big news because it's the first, you know, big model announcement from Thinking Machines, which is, of course, the very high profile startup that was co-founded by the former CTO of OpenAI, Miramirati. um so they came out with this new type of model called interaction models which are you know kind of different from a lot of today's models because they're able to continuously take in audio video and text and then kind of think and respond and act in real time in response to um users that are talking to it um and this is really important because thinking machines has been one of those you know neo labs that we've written about that has raised billions of dollars you know was was valued at$10 billion and then more recently has been in talks to raise at$50 billion, all without really kind of coming out with a really big product.
40:31So this is something that a lot of investors and people have been waiting for for a while to come out. And it's a research preview, right? Which means that it's not available to the public. It'll probably hand it out to a couple of top application layer companies, but it's not available to the public yet, right? Exactly. Yeah. So, you know, we obviously have the demos that the company released yesterday, but we'll have to wait for a bit until developers can actually test out the model for themselves to validate all the claims here. And so the interaction models focus here, the demos on the website are interesting because it's very much, it feels like they're prioritizing this experience of talking to AI or a model that would be used, you know, voice mode is one application.
41:17But I mean, really, I'm thinking about these assistants and even like the models that could be used in, you know, the speakers that you've reported AI companies are working on. So this is really about making AI feel more like a person, right? Exactly. That's definitely a really big part of it. You know, the issue with today's models is that most of them are what people call turn-based models. So that basically means that the model and the user are taking turns talking to each other, and the model has to kind of finish what it's saying before it can really take in new information. So if you use Chat2BT today and try to talk to it, for instance, you might notice that if you try to interrupt the model while it's talking to you to maybe correct something in its understanding or to give it new information, the model often gets confused and will just completely stop talking, might ask you to repeat yourself.
42:10It's just a very kind of stilted and uncomfortable way to have a conversation with the model. That's different from humans where whenever you and I are talking, while I'm talking, you might say like, or yeah, just acknowledge that you're talking. And I don't pause. I just kind of continue talking. And maybe I take in the new information that you're telling me and use that to change the answer that I'm saying. So, you know, all these things are trying to make AI models a little bit more natural and human-like when you talk to them, which is important, as you said, for things like assistance and also devices that you might be communicating by talking out loud.
42:44Or maybe they need to be able to watch videos of the things that are going on around you, if it's like a speaker sitting in your living room, for instance. So I'm having deja vu here because you and I had this exact conversation a couple months ago when you wrote, I believe it was an AI Agenda newsletter talking about how OpenAI has been working on a very similar type of model, no? Yep, yes. This is exactly what we talked about a couple months back. So OpenAI is similarly working on this type of model. They're calling it a bi-directional voice model. Again, AI that you can interrupt and talk to much like how you're talking to humans.
43:23OpenAI actually did release a new update to its voice models last week, which I think some people saw as a way for them to get ahead of the Thinking Machines announcement and try to seal the spotlight a little bit. Those models though got mixed reviews from developers. They were an improvement and they do seem smarter, better at doing kind of real-time translation and transcribing, but they didn't really fully feel like the, you know, what we had imagined when we thought of bidirectional models that OpenAI is working on. So maybe this feels like maybe an earlier version of the eventual bidirectional model that they want to release.
44:02So right now, you know, both Thinking Machines and OpenAI are very clearly interested in this. both of them have not really released the full kind of end product for developers to use. So I think we'll take a bit more time to see who really is going to come out on top with these new types of, you know, voice or interaction models. Right. Great. Well, Stephanie, I want to thank you for coming on. That is Stephanie Palazzolo, our AI reporter here at The Information. That does it for today's show. A reminder, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it then, episodes are available on theinformation.com, our YouTube channel, or wherever you get your podcasts.
44:43Make sure to follow us on social media, on X, Instagram, and TikTok. I am already excited for our next show tomorrow. Have a great rest of your Tuesday. Bye-bye for now.
From the publisher
Deputy Bureau Chief of Finance Cory Weinberg and Mostly Metrics author CJ Gustafsson join TITV Host Akash Pasricha to break down how OpenAI stands to gain over $5 billion from the upcoming Cerebras IPO through unconventional "penny warrants". We then explore exclusive reporting from Aaron Holmes on Microsoft’s renegotiated revenue-sharing deal with OpenAI and how the tech giant has already doubled its $13 billion investment. Next, Rocket Drew provides updates on the Musk-OpenAI trial featuring testimony from Satya Nadella and Ilya Sutskever, followed by Replit’s Michele Catasta on the new "VibeBench" for AI coding models. We wrap with Stephanie Palazzolo discussing Thinking Machines’ high-profile research preview of real-time interaction models.
Articles discussed on this episode:
https://www.theinformation.com/articles/openai-making-billions-just-promising-buy-suppliers
https://www.theinformation.com/articles/openai-save-97-billion-2030-latest-microsoft-deal
Subscribe:
The Information: https://www.theinformation.com/subscribe_h
Sign up for the AI Agenda newsletter: https://www.theinformation.com/features/ai-agenda
TITV airs weekdays on YouTube, X and LinkedIn at 10AM PT / 1PM ET. Or check us out wherever you get your podcasts.
Follow us:
X: https://x.com/theinformation
IG: https://www.instagram.com/theinformation/
TikTok: https://www.tiktok.com/@titv.theinformation
LinkedIn: https://www.linkedin.com/company/theinformation/
Chapters:
00:00 - Introduction
01:13 - Cerebras IPO: OpenAI’s $5B Potential Windfall
14:25 - Exclusive: Microsoft Recoups OpenAI Investment
26:11 - Musk vs. OpenAI: Nadella & Sutskever Testify
30:20 - Replit President on Benchmarking Coding Models
40:21 - Thinking Machines Teases New AI Interaction Model
