Anthropic Accuses Alibaba of Distillation Attack on Claude

0:00Today on the podcast, Meta is changing its mind on reassigning thousands of its employees to AI training. and we also have Anthropic, which is accusing Alibaba of a$28 million query distillation attack on Claude, basically where you ask Claude, you know, 28 million questions, you get them back, you take the questions and answers, and it's called distillation, but you use them to train your model. General Intuition has just raised$320 million at a$2.3 billion valuation to train AI agents on gameplay, and Claude's paying customers are up 75 % since January. This is pretty big because they're starting to eat severely into chat gpt's lead we also have a censure who is rationing ai tokens because their employees are burning them on basic tasks that are not super important and naveen rouse unconventional ai claims a 1000x cut in inference power which again i'm super excited anytime we can cut the water and the power used for ai because that is basically how we make this scalable and we unlock a lot of really cool potential if you've ever been sick of having to pay for multiple subscriptions to multiple AI models and you want to talk to all of the top AI models in one chat.

1:10This is something that always annoyed me when I was stuck with just Claude or just ChatGPT or just Gemini. You know, I wish I could get the responses from others in there, especially when you have something like Claude, which doesn't generate images or you have something like Grok, which might not be as good at generating audio as 11 Labs. I've built the AI Box Playground, my very own startup, where you can talk to all of the top 80 models in one place. So you can ask Grok that question that Gemini refuses to answer. And then you can get 11 labs to create an audio file for you. You can get chat GBT to do deep research and Claude to help you with reasoning.

1:41You can do it all in the same chat. So if you want to go check it out, there's 80 plus AI models, there's image, audio, video, even music models that we have in there. And you can talk with all of them in the same conversation, start multiple conversations, and you can compare the results side by side. Something I love is asking, you know, one model to generate an image and then asking chat GPT to generate the image and you can see the image results side by side pick what model is better for which task you're working on if you want to try it out it's AI box.ai slash playground it's only$8.99 a month and I will leave a link in the description to go check it out okay let's talk about what's going on with meta they have reversed the decision that they made last month to read a pull away thousands of their engineers and they were going to do this to a whole bunch of different AI training roles they had this internal memo that leaked it was seen by business insider and it went very viral but basically it said that they are going to, quote, defer to each individual's choice when they're talking about if they want, you know, people to go and work on the AI effort.

2:33So in the, you know, before they were like, okay, look, we're making a whole bunch of our employees are going to be forced to kind of go and work on this AI training. They were reshuffling how they were organizing their whole company. And I think they got a lot of backlash from inside of the company. Apparently their CTO, Andrew Bosworth, said in a recent interview, he said that it was, quote, probably one of the worst it's ever been when he was talking about the morale over at Meta, just because I think they'd done a bunch of layoffs. They're making a bunch of shifts. And so anyways, they had this big backlash.

3:01They had a whole bunch of people inside the company that were like signing up, petitions against this big move. And so now they're going to let people decide if they want to work on the AI effort or not. My only thing that I will say here, because this was about 7 ,000 employees that were kind of recruited in May that were going to be doing like data labeling and a bunch of other mandatory AI stuff. Um, I'm, you know, they, they have the option to do it or not, but I'm sure there's some percentage if they say no, they just will be let go or something. So I'm curious to see, um, how much of an impact this is.

3:33However, I think just giving people the choice is definitely something that will have a good, uh, like it's, it's good overall for the morale, but I don't know if it's like a really like, Oh man, look, men is just letting them do whatever they want. So I think there's a little bit of that. Okay. Okay, Anthropic is accusing Alibaba of doing 28.8 million questions in a distillation attack on Claude between April and June. They said they had about 25 ,000 fraudulent accounts. This is the largest known distillation attack on the company to date. They said, and they kind of disclosed this in a letter to the Senate Banking Committee, that there is a major loophole.

4:09They said that foreign labs can bypass U.S. chip export controls by simply scraping models' outputs to train their own AI. Okay, there is so much to unpack here. The first thing is they're like, they're bypassing US chip export controls. Okay, forget US chip export controls. What they're really saying here is anybody can do this, like Alibaba, whatever, anyone, and could go train a model, could go make that model open source. But like, what is Anthropics moat if Alibaba can go and run 28 million queries and distill their model? Forget about the US chip export controls. Like, is Anthropics really saying that their model training is so fragile, it could just be copied by doing that?

4:49You know, the billions and billions they spend could just be copied with 28 million queries. And if you think about it, you know, 20 ,000 fraudulent accounts, let's say they're paying $20 for each of these fraudulent accounts, although likely they're paying for the API. but let's just say it was the$20, you know, a month account. We're talking about, you know, $500 ,000, like really? So for$500 ,000, they could, they could pull something like this off. Now I think it's probably a bit more, but I mean, let's just, let's go 10x,$50 million, right? Let's say they spent$50 million in API tokens. Like Anthropic is going to the government, it's going to the Senate Banking Committee.

5:22And it's like, oh no, like these foreign actors are doing this. And I think even Elon Musk admitted in his recent hearing with OpenAI that Grok and and xai had done some distillation like distillation model distillation on open ai so everybody's basically doing this and uh anyways it's fascinating to me that they're so concerned about this because they're spending billions to train these models and someone could come in and spend 50 million let's say do distillation and have a model almost as good that's pretty crazy apparently this particular attack was way bigger than deep seek moonshot moonshot and minimax who all did distillation attacks allegedly to Alibaba back in February.

6:02So this is kind of the biggest one. It seems like all these Chinese companies are like, hey, we'll forget the US chip exports. We're just going to go distill Anthropic. The Trump administration separately ordered Anthropic to suspend Fable 5 and Mythos 5 to all foreign nationals because of national security. But they're saying like, look, you know, we had to suspend Fable 5. But like, let's say they didn't suspend it or even in the amount of time where it wasn't suspended. What if Alibaba went and did a distillation attack on Fable 5. And then Anthropic, you know, the Trump administration forces Anthropic to suspend it.

6:33But now, you know, Alibaba has it distilled and now they have that capability and we don't. All of this makes me quite bullish on open source models in the future because, I mean, perhaps Anthropic's model is just going to get, you know, exponentially better and better for all eternity. But assuming there's ever a plateau, it's going to get distilled and put into an open source model. And people, if they have the right hardware, won't have to pay Anthropic. Although with the price of, you know, memory and everything going up like crazy. If you saw Apple had to boost the prices of basically every single Apple laptop by at least$100.

7:03And for the pros, it was like$300 because parts are so much more expensive and there's such a shortage. So maybe even if we were all using open source models on our own devices, we're all going to get hosed in the future. Anyways, only time will tell general intuition has just raised$320 million at a$2.3 billion valuation to train agents on gameplay. So specifically video game footage. This is a company out of New York. They said on Thursday that Kochal Adventures led their round. They had General Catalyst, Jeff Bezos, Eric Schmidt, a former F1 champion, Nico Rosenberg, and researchers at Google DeepMind and MIT were all writing checks into this.

7:37I mean, if you got the researchers at DeepMind and MIT, you probably know you're onto something pretty solid. This is a pretty big round. They actually did a round of funding back in October of last year for$134 million. So total, they've raised$454 million. They're kind of a famous company because General Intuition fine-tuned a robot using eight minutes of real-world data. And then after pre-training on gameplay from their company Metal, they were, you know, the robot was actually able to walk and move. But they're using this data from like video games to help robots in the real world. And this is where they're kind of becoming, you know, this is what they're getting famous for.

8:15The reason why this one in particular is exciting is because if this works, then general intuition is going to be kind of the model layer that runs a whole bunch of robotics and simulation and gaming startups. So the same way that Anthropic and OpenAI and you know, all of these kind of general models are running all the startups today that do software, this will be doing all of the robotics startups. So that's why it's exciting. Claude's paying customers are up 75 % since January. This is a huge jump. This is according to a bunch of transaction data from 28 million US consumers. And this is the first serious jump.

8:50And I guess you could say competition to chat GPT subscription dominance, because right now everyone knows, okay, look, Claude's doing really well with, with enterprise. There's a lot of enterprise players paying Anthropic a lot of money. And I think it, you know, slightly is beating out opening eye there, but opening eye has always been the king when it comes to consumers, right? They're the ones that have the cheapest model, they came out first, they have kind of the most usage, but Claude is getting more and more popular with that demographic as well. If you go look at one of the indicators when people are talking about these metrics is there's, I guess, a bunch of like courses.

9:22And so these are kind of self-directed learning courses. And if you look at those, the Claude course demand is way higher than ChatGPT. It's actually three to one. So there's a huge shift. And I guess you could say, well, maybe it's because it's newer and so many people already know how to use chatgpt now they need to use claude and to be fair i think claude has probably more to it for the average person right we're all used to talking with chatgpt i'm not going to go take a course on that um although maybe some people will but when it comes to claude and it's like well there's claude co-work and there's claude code and there's like all these different ways you can integrate it into the the you know building tools there's a lot of courses that i think people would want to take and so i think this is probably why it's spiking so much apparently if you go to data camp they said that their users have searched for the keyword Claude more than the keyword AI.

10:05And they said that the demand for Claude courses is up 18x in the last 30 days. So this is kind of a really big trend. And if we look at even the usage of, you know, enterprise, we're seeing just a huge upswing. So ChatGPT is still the leader in kind of the absolute amount of paying customers that are, you know, using their model. But there is some really big growth coming out of Claude. Accenture is now rationing employees' AI token access because apparently a whole bunch of their staff burned through all of their budgets, converting PDFs to slides and running a lot of other really low value tasks through these super expensive models.

10:41Now, it's interesting, and this is something I talk about a lot of my show, just because you can do something with AI doesn't mean you should do something with AI. There's a lot of tasks where we just have very basic software that we've had for a long time and it gets the job done. You do not need to use AI to do it. This is one of the cases, right? Converting a PDF to slides. There's probably a lot of other ways to do this that are using the most expensive frontier models. And Accenture learned this the hard way. This is something that is, I guess, kind of reversing a month's old mandate where they were threatening people and saying, hey, look, you can't get a promotion if you don't adopt AI.

11:13And now everyone uses AI for everything. And now all of a sudden they're like, oh, shoot, like this is actually pretty expensive. There's not a lot of ROI on some of these tasks that people were doing. So they're kind of walking it back and changing their mind there. Justice Quack, who is the agentic AI strategy lead, told executives that unpredictable month-to-month AI spend makes it nearly impossible for finance teams to model costs, triggering CFO, COO, and CIO level scrutiny. Earlier this year, Accenture built internal leaderboards where they're basically ranking their employees on AI usage to drive adoption.

11:46This obviously looks super counterproductive today. just saying whoever burns the most tokens on, you know, and maybe that could be on really low value work shouldn't be the people that are being, I guess, compensated the most or promoted the most. So right now they're now switching gears. They're going to start rationing. And this is also coinciding with a multi-day AI stock sell-off, which is hitting a lot of memory chip makers. I think this shows that a lot of enterprises are unable to justify some of their inference hardware spending. A story I'm super excited about is that Naveen Rao, so he is the ex-AI chief at Databricks, he left and he launched his own company, which is called Unconventional AI, and he said that an oscillator-based chip architecture can cut inference power by up to 1000x versus a GPU.

12:33So he has this new technology, and he believes it is 1000 times more product or more, you know, efficient, power efficient than a GPU. They just released their first un-o, which is a working image model. It's basically matching stable diffusion's quality. This could be built at the perfect time when we're basically facing a massive crunch of not just energy, but just, you know, having enough compute power with all of these hyperscalers. Sorry if this sounds a little bit technical, but oscillator computing uses coupled physical oscillators to settle into low energy states. And they're doing this instead of what we're we're typically used to with GPUs, where they're shuffling bits through logic gates billions of times a second.

13:13So we're completely changing the way that we're doing the architecture here. And I think we have a real need because hyperscalers have this huge power wall, right? All of the big frontier labs, OpenAI, Anthropic, Google, they're all signing, you know, nuclear PPAs for gigawatts of generation. And all of these, you know, they're trying to get the energy generation from all of these new energy facilities. But it takes so much time to build not just the data centers, but also a lot of the energy generation, right? The nuclear deals that Microsoft is signing, all of that isn't going to arrive till the late, you know, the late 2020s.

13:43And that makes, you know, basically any real efficiency gains that we can make today huge, because that ties us over until a lot of these new power plants come online, a lot of the new data centers come online, we have a lot of, you know, infrastructure that's going to get built out. And we like, we're going to run out very quickly, I think everyone's realizing that. And so when we have these efficiency gains, it means a lot for the industry. So very exciting. Guys, thank you so much for tuning into the podcast and make sure to leave a rating and review. So if you're on Apple dropping some stars on Spotify, it really helps a show out.

14:13It basically boosted up in the algorithm. It helps more people find it. And it helps me out a ton. Also, if you have topics that you really like that we cover, or you want to hear more about something, make sure to leave it in a review. I read all of the reviews. I read all the comments and I really appreciate them. And I help, I try to let those help guide the show to improve the quality on it. So if there's anything else you guys want, let me know in the comments and I will catch you guys all in the next episode. Make sure to go check out AI box.ai as always, if you want to try 80 different AI models in one place for$8.99 a month.

14:42All right. Catch you on the next episode.