OpenAI & Anthropic Cost Crisis, Superhuman's AI Slop Fight & SpaceX vs Tesla Hype

22:52Right.

22:53Stephanie Palazzolo:Let me ask you one question on a separate topic here. We were talking in a segment before you about how companies are grappling with the rising costs of using LLMs. And I know when you were on our show, I think both times actually you were talking about how many API calls Grammarly, Superhuman does. I think it was something like… Over 100 billion a week. Hundreds of millions or billions. Anyway, how are you grappling with LLM costs and using these models? Are you looking to open source models more at all and integrating those into your stack? Yeah, so we do about 100 billion LLM calls, well over that now per week.

23:38Probably a more interesting stat is per user. That works out to a few thousand per day for every user. That means for any user of our products, especially the Grammarly products, we are likely your number one AI generator. We're probably generating more LLM queries on your behalf than any of the other tools that you're using. So it does put us in a very interesting spot of the flow of AI. About 97 % of those calls go to in-house LLMs today. And about 3 % go out today.

24:07Stephanie Palazzolo:That you have, that superhuman has. That we host and train. Yeah. Okay. A lot of them start with an open source starting point. The Lama models are quite popular. The Gamma models from Google are also quite popular. So we will start with them. We'll fine tune and train them, and then we'll serve them from our own infrastructure. And that allows us to run at much better cost. The core of those products run at 85 plus percent gross margin, which is obviously very different than some of the other AI providers out there. I think that pattern of find use cases that work at scale and then take general purpose models and then reduce down to models that are more specific to those tasks will be a common pattern.

24:50Because I think users want, as a customer, you want AI that works where you work. You don't want to go to a destination. Maybe you go once a day. You want something that really feels like a companion that works right with you. And every time you type a character, every time you open a new application, whether you're in anything from Gmail to Slack to Apple Notes, you want that set of agents working right alongside you. That can be expensive, and so you have to find ways to scale that. And a big part of our effort has been on making sure we can deliver AI at scale, at low latency, and at reasonable costs.

25:30Stephanie Palazzolo:97%. So what was that this time last year? Was it that high? roughly. You know, I'm just trying to get... In the same ballpark. Yeah, we've been... Okay, so you've always been building your own models. Grammarly is interesting. Grammarly jumped on the transformer bandwagon before GPT-2. And so the team has been working on this for a very long time. So the building our versions of these models so that we can deliver AI at a fairly high scale, especially as you think about it on a per-user basis, thousands of calls per day. And you have to be able to do that with reasonable costs and latency. I mean, probably one of the most important things for us actually wasn't the cost.

26:10It needs to work as you type. So as you're typing characters, we need to be constantly looking at every word and saying, what are the different things we can do for you here? And we have to do that really fast. And doing it fast ends up leading to an architecture that also lays a better cost as well.

26:27Stephanie Palazzolo:What about your own employees, the models that they use for coding, for example? I imagine your tech team is using all these frontier models to expedite their own processes. I mean, have you burned through your own budgets for using these frontier models? And have you told your CFO that we got to keep a tighter lid on things? So far, and we use, just to be clear, 97 % of our customer traffic goes to our own models, 3 % goes to the others. That's still a huge amount. So, you know, we are big customers of all the big LM providers internally as well. I mean, we're right at the bleeding edge of all the different tools, pretty heavy AI adoption amongst all our teams.

27:12But certainly the engineering teams are pushing the limits on it. You know, my mind, I think the companies that are worried about this are measuring the wrong thing. And certainly if you start with a view of just looking at token spend as your measurement, and I think now we're seeing a lot of companies that are backing away from token maxing as a way to look at it, you'll get all the wrong incentives. Instead, we have a set of different metrics we look at to evaluate whether we're getting ROI on our token spend, especially in teams like our engineering team. We're looking at how fast features are shipping.

27:46We're looking at the throughput of our pull requests, looking at latency and so on. And all those numbers are way up right now. And so far, I think our ROI and our token spend is incredibly positive. I haven't asked them to slow down at all. we haven't hit our breaking point yet.

28:01Stephanie Palazzolo:Got it. And when you said measuring the wrong thing, you meant the token maxing measuring. That's what you meant. But if you're the type of company where you spend a bunch of AI tokens and you don't get better product or more products shipped to customers faster, then yeah, you're probably wasting your energy. And I'm sure there's people that are in that situation. That's not what we've seen so far. We are delivering product. In the last quarter, we've roughly doubled the pace of feature delivery from our engineers, which, you know, if you just go look at that and compare with the cost of the engineering team, it's incredible ROI.

28:37So I don't think we've hit diminishing returns on that yet. Great.

28:41Stephanie Palazzolo:Well, Shashir, always a pleasure to have you on. Thank you for joining us. That is Shashir Mehrotra, CEO of Superhuman here on TI-TV. Okay. SpaceX shares and Tesla shares are interesting to watch on their own, but they are even more interesting when you compare them to one another. My colleague Theo Waite wrote a column about the extent to which people are selling Tesla to get into SpaceX. Maybe they're doing it the other way around, too. I want to bring him on to walk us through his thoughts. Theo, welcome back to the show. It's great to have you here. Good to be here. Okay, so what do you think?

29:13Stephanie Palazzolo:Are people selling Tesla to get into SpaceX these days?

29:17Theo Wayt:I think it's hard to say. I mean, if you look at the stock price each day, it kind of fluctuates. way it's it's a weird situation because you know in the stock market there used to be one way to buy exposure to elon musk and that was tesla and you know obviously now there's two and they trade in different ways and it's hard to choose between them but presumably one of them is going to do better than the other you know over some time span um so you know clearly all the hype is around SpaceX right now because of the IPO, but that doesn't mean it's actually, you know, a more compelling stock at its current price.

29:57Stephanie Palazzolo:It's kind of interesting, though, because Tesla has its own moonshot ambitions here, right? I mean, they might be a little bit closer term than the orbital data centers, but it's sort of like, take your pick on which long horizon mission you would like to invest in.

30:16Theo Wayt:yeah i mean both of the companies have like revenue multiples that don't that wouldn't really make sense for companies not led by elon musk and like there have been a lot of people for many years that have said that tesla is overvalued but if you look at it it's it's only trading at like 14 times it's 2025 revenue whereas spacex is trading at more than 100 times it's 2025 revenue So if you think about, you know, the Elon Musk premium of, you know, being able to sell investors on, you know, super long-term moonshots, you know, SpaceX has way more of that baked in at this point than Tesla does. But if you think about, you know, six months ago, Elon Musk was saying that Tesla is going to build robots that are going to cure poverty and make working optional for everyone on Earth.

31:08Theo Wayt:That's not exactly like, you know, a small total addressable market or vision or, you know, however you want to think about it. Like both of these companies have gigantic ambitions and one of them has that reflectance thought price more than the other at this point.

31:23Stephanie Palazzolo:Where are the Optimus ambitions at for TestLad right now? Bring us back to Earth on this.

31:30Theo Wayt:Well, there's this V3 version of Optimus that's supposed to be the new one that addresses all of these issues that we've written about. You know, last year we had some good stories about how the hands are— You. You had some good stories. Well, and Rocket. And Rocket. Okay, and Rocket.

31:48Stephanie Palazzolo:The hands one was Rocket, that's right. I don't want to erase him here.

31:52Theo Wayt:um he is is you know an expert on this too but anyway uh you know they they've been trying to work out a bunch of kinks and optimists and those uh you know they've not revealed the um new version that's supposed to have addressed all those problems and and that keeps getting pushed back but elon says they're still um you know they're building a new factory and they're putting up all these assembly lines and they're gonna be ready to start mass production whenever they have something to mass produce. So that's still, you know, to be seen, but it's, Elon says it's still a big part of the vision for the company.

32:32Stephanie Palazzolo:So the arbitrage play here that you hypothetically lay out could be available to investors is what? Basically sell your Tesla and buy SpaceX.

32:47Theo Wayt:as soon as possible to be clear this is not financial advice i know yeah i do not trade individual stocks i'm you know whatever take this with a great assault but in my opinion uh you know both companies are are you know valued in super extreme ways but the spacex one is extremely you in another orbit compared to Tesla's. And Elon and Gwen Shotwell and other people have been, you know, dropping hints for a while now that SpaceX and Tesla could merge and have this super Musk company that is, you know, all of his companies rolled into one. And if you think that's going to happen, you have a limited window to, you know, get into either side of that merger if you want.

33:39Theo Wayt:And, you know, in my view, it seems more likely that the amount of hype around SpaceX and the relative lack of hype around Tesla means that it's probably more likely that SpaceX will go down a little bit in multiple and Tesla will go up a little bit and they'll meet in the middle somewhere. And so if you're someone that thinks, like, I want a certain percentage of my portfolio allocated to Elon Musk, and you're trying to decide between these two companies, maybe Tesla is slightly less hyped and makes a bit more sense to go for at this moment. And you're not going to have, probably, you're not going to have the opportunity to buy into Elon, Inc.

34:18Theo Wayt:in so many different ways in the future.

34:21Stephanie Palazzolo:Unless perhaps the boring company decides to go public at some point. That's another no. You never know. Hey, before you go, I do want to ask you about this debt offering that SpaceX made public yesterday. They started on this debt offering. Are you expecting there to be more debt offerings of this kind in the next couple months and years?

34:45Theo Wayt:Yeah, I think so. So, I mean, you know, I think they have something like$100 billion in cash right now. But if you look at their ambitions for, you know, data centers and space and the moon and all that, you know, not to mention the amount of money that they're currently burning on XAI's data centers, you know, they need a ton of cash. And, yeah, I think if, you know, they're able to raise, you know, more debt, why would they not? I think they definitely will.

35:15Stephanie Palazzolo:Great. Well, Theo, I want to thank you for coming on. That is Theo Waite, our Elon Musk reporter here at The Information. The latest AI term to pay attention to is AI agent loops. My colleague Stephanie Palazzolo unpacked what those are all about in this morning's AI Agenda newsletter. I want to bring her on to explain it to us. Stephanie, welcome back to the show. What is an agent loop, Stephanie? So an agent loop is essentially a new way of running AI models or AI agents to get them to work better on longer running, you know, sometimes more vaguely defined tasks. So, you know, I guess to break this down a bit, the kind of old way of giving an AI agent a task is, you know, you tell it, hey, I want you to try doing this thing.

36:02Then you see it attempt that sort of task, you wait for it to finish, then you kind of sit around and give it feedback and give it a new prompt. So it's obviously a very kind of like time intensive manual process where the human user has to be super involved. And so instead of doing that, a lot of people are using agent loops now, which basically, instead of, you know, giving the agent each individual prompt, you instead give it a more higher level goal of something that you want it to achieve. And then you let the agent basically run in loops, trying different approaches to completing that task until it does so, or until it hits some sort of like predefined kind of stop milestone.

36:40Stephanie Palazzolo:So can you give us an example of this? So instead of giving an individual prompt, you give it a higher vision task. So what would be the alternative prompt, I guess, that you would give the agent to sort of encourage this loop? Yeah. So I think one example that you could think of is, you know, let's say you want an AI agent or a group of AI agents to take a look at all your monthly expenses and income and kind of come up with like a budget or a summary of your expenses for that month. So the old way of doing things, you might kind of say, okay, hey, here, first, I downloaded this Excel from my credit card of all the expenses.

37:19I'm going to give that to you. Could you put that into a spreadsheet? And you're kind of going back and forth with the agent saying, okay, now that the spreadsheet is done, I noticed you messed up this one part. Now, could you take that spreadsheet and add my income to it? So it's a very back and forth process versus with this kind of agent loop approach, you might just give it a higher level task. Like, hey, can you take a look? Can you connect to my, you know, or like here's like a folder full of all the documents that you need. Now take all this and just come back to me with a fully built out Excel spreadsheet that has, you know, my income and my expenses and maybe give me an analysis of my month's spending.

37:57And so the agent itself will then kind of go back and forth and do everything it needs to do until it comes back to you with the finished product. Right.

38:10Stephanie Palazzolo:I mean, I feel like I see it. This is not the agent loop, but I see Claude, for example, those thinking steps. You can, if you're really nerdy, you can click the dropdown and see all the intermediate steps that just a chatbot goes through. Basically, this is the agent going through all the intermediate steps in the background and going back and saying, well, this is not good enough. you know let's let's keep iterating on it until we we get a better final product yeah and um a really important part of this process is rather than just having like one agent that is trying to do everything um whenever you give this kind of high level goal to the agent it can actually spin up these like so-called like sub sub agents that are all working together and each have kind of a different role and so for instance a very common kind of framework that people use is to have like a planning agent which takes the very basic prompt that i might give it and then it kind of breaks that down into kind of steps that it will take so if i tell it oh make a budget it's gonna say okay let's break that down first take you know first take the credit card statement and then put that into excel then after that take like you know then break that down and categorize it into these different spending categories so that's like kind of the first planning agent then you might have like another agent that is like the evaluating agent so that's the agent that's kind of like checking the work of the first agent and saying like, okay, is this Excel like breaking anywhere?

39:39Is, you know, are these, are these columns adding up to the right number? That sort of thing. So you kind of have almost like a working agent and then like a checking agent. That's like checking agents.

39:51Stephanie Palazzolo:How do I, how do I start, how do I declare that I want the agent to use the agent loop? Do I have, because I mean, agent loops, this is sort of a newer type of infrastructure that's being used right so do I have to queue it or what yeah so already there are some kind of like agent tools like codex and cloud code out there that are um introducing features that make it easier to run these agent loops and so both codex and cloud code have this new feature they announced in recent months called goals and so what that means is um you can basically give your your agent a higher level goal um like you know speed up how quickly this app runs and then the agent will kind of run in loops trying different ways to achieve that much larger goal so you know we can already see this like agent loop approach being kind of productized by these different companies because they're coming out with features that will make it easier for for a user to to do a loop without having to say you know without having to go in themselves and be like okay i want you to spin up a planning agent i want you to spin up a evaluating agent and have to do that in a more manual way.

40:57Stephanie Palazzolo:I imagine it's more expensive though, right? If the agent is doing so many cycles in the background? Yeah, it definitely is. So one example that actually Anthropic engineers talked about is this app that they made where, you know, doing the more simple approach where you just kind of give it like a single prompt and then let it just try, you know, a single time to try to build the app. That took like 20 minutes and$9. But using this agent loop approach, it took them six hours and$200. So obviously that is like many, many more expensive and also takes a lot longer, but the end product was a lot better.

41:32And I think the good thing with this too is, as each new model comes out, it actually does become a lot more efficient and does kind of help shorten the time and therefore kind of lower the costs of running these agent loops.

41:48Stephanie Palazzolo:How widely are agent loops being adopted right now? Is this something that every company is working on or just the cutting edge researchers? So, I mean, I would say it is still more on the cutting edge in terms of like, you know, I think the average person who's using ChatGPT as kind of a Google search replacement might not really care or need to use agent loops, but you could definitely see it popping up. You know, first at the AI labs, we see people like the Claude Code creator talking publicly in just the last week or so about how he's really been using agent loops a lot. And that's really, you know, lowered the amount of time that he needs to spend coding himself.

42:27And then from there, we kind of see it spreading to, you know, developers or maybe like startups. But I definitely see eventually this making its way into all sorts of, you know, larger companies or maybe slower companies or slower industries that take more time to kind of catch up to the frontier.

42:43Stephanie Palazzolo:Well, and I certainly see this as being somewhat related to the discussion that we had last week on the show about how agents will basically try to improve themselves over time. Feels like maybe if they're going through these loops, then maybe there's a way to integrate those tools in the long run. I don't know. Yeah, yeah. It seems like a lot of this is moving towards, you know, just agents that are able to run by themselves for much longer than in the past. Improve themselves, buy themselves. Exactly. Yeah. And then combining that with what you're talking about, which is the ability to kind of like improve.

43:21And, you know, once you try to task once, make sure that the next time that you do it, you're faster, you're better. So all very promising stuff, I think, for the world of AI agents.

43:30Stephanie Palazzolo:Promising, but before we let you go, is there a flip side to this? Are there limitations still that we should be paying attention to with these agent loops? Yeah, I mean, there's always limitations to all these sorts of things. I think the biggest one today are the high costs and how long these take. And also, one thing that a lot of people warn about is that if you aren't careful and you don't give the agent specific instructions on when to stop. So let's say, you know, you might tell it, hey, try this task 10 times with an after that stop and then give me like an update on how it's going. You can imagine that, especially because these AI models are getting really good at not giving up, it might just kind of run continuously in the background for hours and hours without you even knowing and rack up this huge spending bill.

44:16And then maybe whenever you go check on it, you find out that it actually messed up on something after the second loop and every loop after that has been wasted. So I think you just need to be extra careful in these cases to like, you know, make sure that you are keeping track of the agent, making sure it's not spending too much. So that's been the biggest kind of worry or concern that I've heard from developers that are using this.

44:37Stephanie Palazzolo:Yeah. I mean, anecdotally, again, these are just basic chatbot experiences that I'm referring to here. But, you know, whenever I ask Claude or ChachiBT to stop because, you know, it's having some issue with the prompt or maybe it's taking too long, it never stops.

44:54Theo Wayt:Like, I swear it just keeps going.

44:56Stephanie Palazzolo:And, you know, I have to refresh sometimes. And I don't know. It's just not very good at turning off, I find. So I believe you when you said the agent will keep going despite your hopes. All right, Stephanie, I want to thank you for coming on. That is Stephanie Palazzolo, our AI reporter and author of AI Agenda here at The Information. That does it for today's show. A reminder, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it then, episodes are available on theinformation.com, on our YouTube channel, or wherever you get your podcasts. Make sure to follow us on social media, on X, on Instagram, and on LinkedIn.

45:34Stephanie Palazzolo:I am already excited for our next show tomorrow. Have a great rest of your Tuesday. Bye-bye for now.

45:44Thank you.