Arena AI hits $100M run-rate in 8 Months

0:00Arena AI has just hit a$100 million run rate. This is only eight months after they launched paid evaluations. I want to break down what Arena does, why they're special, why they're unique, and why they're growing so fast. Palantir is tapping NVIDIA's Nemotron open models for US government AI. This is interesting, and a lot of drama is behind this story as well. Elizabeth Warren and Scanlon are reviving a bill to ban AI firms from selling health data. We'll get into the details on that. Flexion Robotics is training hundreds of humanoids to run office errands autonomously. And China's CXMT is landing a$3 billion memory supply deal with Tencent.

0:42We know memory is one of the critical pieces of the AI infrastructure build out. And there's a lot of issues going on with memory increasing the costs of basically all electronics today. If you've ever been using an AI model like Claude and have been frustrated that it doesn't create images or audio or video, I'd love for you to try the MCP connector for AI box. That's my own startup. We essentially allow you with one link, give it to Claude and it can bring any of the AI models that you use for everything else into Claude. So you can use chat GPT's image generation inside of Claude. You can use 11 labs audio generation inside of Claude or Google VO three's video generation inside of Claude.

1:18So all of the capabilities of the different AI models, there's 80 different ones that we allow you to connect. You can bring them all inside of Claude or chat GPT or Gemini or cursor or any of your other places where you really do all of your work, all of your workspaces with AI. So if you want to check that out, it's AI box.ai slash MCP. It's a super easy MCP connector. So you give Claude this link inside of the connectors and you log into your AI box account. And now you have all of these capabilities. I have a whole website where I explain how this works, how you can do it. And it is$8.99 a month to get started with it.

1:48So it's super cheap. And I hope that really unlocks a lot of creativity for you to be able to get 80 different AI models and capabilities inside of Claude or whatever else you are building with. Okay, let's talk about what's going on with Arena. This is a company that is now at$100 million. They were originally created in UC Berkeley, and it's basically an AI leaderboard, right? Like this is the company where you can go and test different AI models against each other. They've hit this$100 million annual revenue, and this is eight months after they launched their paid evaluations just back in September.

2:20This is triple what they were doing in January, which was$30 million, which honestly, even in January, where I thought this was really impressive. So the company is now directly competing with Scale AI and Merkur for post-training dollars because they're selling labs structured analytics built on 10 million plus human model comparisons. So basically what's going on is they've just raised $250 million total across two different rounds. So they did$150 million Series A in January at a $1.7 billion valuation. They did that from A16Z, Clymer Perkins, and Felicious. So the way that they actually are making money is that the public leaderboard is still free.

2:55So essentially, if you haven't tried this before, you go to the site and it puts two different AI model responses side by side. A lot of people use this because you basically get free AI usage out of it. Instead of having to pay for ChatGPT, you can go pay this and it will give you, you know, two responses side by side. You pick which one you like better and you're helping, you know, tell it which ones are the most popular. Those leaderboards are then, you know, created so you can see, you know, oh my gosh, ChaiGPT's new model is, you know, beating the open, you know, the model from Anthropic, whatever.

3:23So that's kind of where a lot of these leaderboard companies, how they kind of run and why people use them. But how they're actually making money is that they have an AI evaluations product package, which is going to show data basically to different AI labs. So like OpenAI, Anthropic, and Google, when they have their models in there getting voted on, they'll show them what areas their models are losing on to competitors, right? So it's like, hey, look, your model's good, but anytime a healthcare question gets asked, yours is doing poorly. Or anytime a finance question or anytime a question related to this.

3:55So they're telling them exactly where their models are lacking, and they're helping them to guide reinforcement learning. So it's a massive value for OpenAI and Google. They're going to pay a ton of money for that. It's a great value for people that get free AI usage. And also, it's interesting for all of us to see what companies are doing the best in the leaderboards. So Merkur's annualized revenue, which is one of their competitors, hit$1 billion this year, and Handshake AI's training army grew from$550 million to about$1 billion in three months. So I think this just kind of shows the scale of a lot of these post-training markets, how much money they're actually able to make.

4:30Let's talk about what's going on with Palantir. So they have just basically selected NVIDIA's Nemotron open models to use at the U.S. government for all of their AI. Of course, there's a ton of drama with Anthropic and even OpenAI right now and the U.S. government having to, you know, pull their models or tell them, hey, you can't release your model till we give it like an accurate assessment, whatever, right? So one thing that's interesting, I think, in particular with the Department of War and a lot of the beef that they had with Anthropic and them, you know, classifying Anthropic as a supply chain risk because they wouldn't let them do all the things that they wanted to and stuff.

5:02Because of that, Palantir, who does a lot of the development for the U.S. government, a lot of software development, has selected to use an open source model. So NVIDIA has their Nemotron model. They're using this open source model, and they're using it to run a lot of stuff. Now, a lot of AI work inside of the U.S. government. And the reason why an open source model is great for this particular use case, or why the U.S. government would be so happy to use it, is because there is no centralized servers that the model's got to go back to. there is no one that can say hey you can't use it for xyz reasons it's an open source model you can do whatever you want with it so um in basically everything that they're doing they will have uh they won't have to go and get it signed off i remember there's kind of this famous conversation between um the the head of war and um the ceo of anthropic uh dario and they were saying like hey we need it to um to be able to do xyz tasks and dario was like oh well um if you're going to do that like technically that's against our terms of service and then they're like okay well what if like there was a moment when we really needed to make some sort of, you know, critical decision on this.

6:03And Dario was like, Oh, well, you could just call me and like, I could if there's like an exception to like a rule, just call me and like, I'll unblock something for you. And they're like, Okay, well, we don't want to call you and unblock something. And for the middle of like a confidential, you know, battle thing, blah, blah, blah. So I mean, a lot of people have a lot of opinions on this. But these are the two arguments that we hear on this. And so this is why it looks like Palantir is is going to be using this to run a lot of US government agencies. And beyond all of the drama. I just think for keeping classified data without having to send that to an outside network, I think a lot of agencies will want to use that.

6:32So agencies also get to keep the customized model weights that they train, which I think solves basically the core problem that's kept a lot of secure AI adoption stuck and being very slow in the past. And in particular with Nemotron, this is an open model and it's going to run inside of Palantir's Sovereign AI operating system. It's going to be all run on NVIDIA's accelerated hardware. There's going to be handling data authorization, there's going to be isolation, and all of the auditing is all going to be in a closed loop. I'll also say this isn't too crazy, like two thirds of companies right now already use open models.

7:05And I think a lot of them say they're doing this because of cost efficiency. Nvidia frames this as kind of the template for scaling AI in regulated environments, sensitive environments, so like finance and healthcare and government, obviously. In AI regulation, Elizabeth Warren and Scanlon are reviving their Health and Location Data Protection Act. They're adding a bunch of new things to it. It now explicitly bans AI companies like OpenAI Anthropic and XAI from selling health and location data users enter into Chai Chapti, Claude, and Grok. The bill is going to allocate$1 billion to the FTC over 10 years to enforce this.

7:42and basically this is filling a regulatory gap because a lot of these AI labs are racing to get users and I mean even myself you have chat GPT they just added chat GPT finance which of course I my wife like hates this kind of thing she hates giving data to AI models and it's probably for good reason myself though I went and added basically all my credit cards and bank accounts to my chat GPT it does the integration through plaid but it's been super useful to be able to ask it questions about where we're spending, what are recurring expenses are, income from businesses. I mean, there's just so much interesting data you can get just chatting with this thing.

8:19So I love it. I know a lot of people use this for health things as well. And there's a lot of benefits, but a lot of people are concerned about, you know, giving all of these financial and health data to these companies, what are they going to do with it? So the FTC has to write implementation rules within 180 days of this bill being passed, if it is passed. The agency, state, and attorney general and individuals can all sue for any violations on this. There's some co-sponsors on this. We have Senator Ron Wyndon, Bernie Sanders. The original bill is from June of 2022, but it only covered data brokers.

8:53It didn't have AI companies and kind of how they're collecting data upstream. Chatbot health transcripts are unusually rich, right? Because users are going to be pasting in their lab results. Are they going to upload scans? They're going to ask follow ups, all of that. And all that can create a lot of really complete medical profiles that no other health, you know, traditional app would actually be able to capture. People are uploading pictures of their x rays and so much more. So anyways, I do love all of this kind of data protection, not being able to sell that that makes a lot of sense to me, even though I don't agree with everything that Bernie Sanders does or says when it comes to AI in particular, banning AI data centers from New York.

9:35I mean, I just think that that's kind of counterintuitive to a lot of what progress and the benefit that these data centers and well, the benefit that a lot of this AI can have on people. But anyways, all of that aside, this is one where I would definitely say I'm agreeing with Elizabeth Warren and Bernie Sanders on the face of this bill. Now, I don't know if there's secret sneaky things stuck in there. But overall, don't sell my health data to outside companies that I give you, I am definitely in agreeance with that one altogether. Okay, let's talk about what's going on with Flexion Robotics.

10:07They are training humanoid robots to run office errands, they're helping them do this all autonomously. This is a Swiss startup, it was founded by ex NVIDIA researchers that built a software stack, essentially, that's letting these humanoid robots do multi-step office tasks. So things like, and oh, and by the way, you can do this all by just talking to them. So I think I'm really used to seeing those like dog robots from Boston Dynamics and there's like remote controls where you can kind of control them. I've seen them at like conferences and stuff. This will be different. This is, you're literally going to humanoid robot and you're like, Hey, go get me a coffee.

10:40Although I'm sure there's, you know, better things than telling it to go get you a coffee. It'd probably be like walking around and getting things faxed or scanned or dropping things off different people's desks. I don't know, right? This could actually be quite useful. I almost imagine this like if I was in an office setting, everyone would have like a little microphone button maybe that like summons this thing. And you could like think of how you kind of use like Claude or I guess how I use Claude is I hit like my voice to text button, I give it an instruction and it runs off doing it. Let's say there's one humanoid robot for the entire office.

11:11Not everyone needs their own, obviously. So you could have like one or two, maybe one per floor or something like that you push a button you give it its command you tell it what you want it to do it comes over to your desk it grabs the document grabs the paper grabs the you know whatever you need it to move or do and then it goes and executes it it's like a real world version of Claude I love this if you can't tell basically this is a modified unitry humanoid robot and it's running all of their systems it's um you know they're getting it to like go retrieve like mail and stuff. Apparently it can go up and down stairs.

11:42It can go in the elevator. It's super capable and they have it stocking a shelf in a recent demo that they released. So it's very capable. Right now they're using reinforcement learning at basically every layer of what they've built. They have a master planning, they have simulation environments, they have motor controller. If you look at the overall market, ABI research is projecting that robot foundational model markets are going to hit about$150 billion by 2036. Flexion is basically trying to position some of the software as the product. And they're saying, look, the hardware is a commodity, there's gonna be a ton of these companies that are building these humanoid robots, maybe Tesla, we're not going to compete on there.

12:20But we're going to compete on the software that helps train these robots to do things in your environment in your office, and how you guys actually control that. So honestly, I kind of love that. It's moving, it makes you know, the humanoid robot feel more like an LLM and there's all of the stuff getting built on top of it. Flexicon is hardware agnostic so they're saying they work with a whole bunch of different humanoid robot makers. They're not going to build their own robot. They're going to be able to let you use Unitree, Figure, X1 and a bunch of other platforms. China's CXMT has landed a three billion dollar memory supply deal with Tencent.

12:54This is the largest commercial contract I think in basically domestic DRAMs makers in like their entire history of their company. The deal right now, I think, is showing that China's largest cloud operators are now actually willing to source AI memory from local suppliers, because a lot of the US export controls are tightening around basically all of the advanced chips, right? So Tencent is a huge customer, they need so much of this. And traditionally, they have bought a lot of this from American companies. But it seems like they're able to get this from Chinese companies now because of these export controls.

13:26Something interesting about this company in particular is that CXMT is kind of moving from like smartphones and PC OEMs to now hyperscale AI customers. And I think that's kind of the first time for them. It's very similar to what SK Hinnix did. And they kind of, you know, hit this dominance path because they were getting customers like Nvidia to buy from them. I'll be interested to see if CXMT is able to move outside of just China and really get more global dominance, or if it's something that's just going to be geographically landlocked because of kind of the sensitive nature of everything happening with cloud and AI and and all of the data center build outs.

14:00This podcast was a lot of fun today. I'm actually sitting here on a yoga ball holding my baby. I got a new baby as of a couple weeks ago. And hopefully you weren't able to hear him grunting too much. He's sleeping. So he should be pretty happy. But anyways, I'm gonna keep getting these podcasts out for you. There's so much happening in AI news. I love this stuff. I'm super passionate about it. But lots of fun going over but excuse any uh little sounds you might be hearing in the background um as uh we're recording in the studio with clay so anyways if you wouldn't mind leaving a review it would help to show out a ton i appreciate them all i read them all um but anyways thanks so much and uh make sure to check out ai box.ai the mcp in particular if you want to get access to all of the different ai models there's about 80 of them in one platform or you can build them all into clod so you get image audio video all inside of Claude.

14:51All right, you guys already know the links in the description. I'll catch you in the next episode.