AI Labs Debate Safety, Suno V6 Arrives & Bending Spoons’ Miro Deal | Lightwork

0:00The part that I loved about Robbie's piece was that he specifically says the founders worth backing hold convictions before they're obvious. It's interesting that he's bringing it all back to principles. It feels like everybody is identifying the problem, but nobody can really agree on the same solution. Right. Spending Spoons is acquired yet or in talks to acquire yet another company. Did you see that Suno v6 launched? For me, this is just like the dawn of a new breed of creator. I've actually had a deep fake version of me. Already?

0:38We're back with another big week. Daria Amadei called on the AI industry to pace the frontier, with Sam Altman saying yes and Jensen Huang saying no. Suno shipped the first AI music models built with a major label. Plus, Bending Spoons agreed to buy Miro five weeks after Airtable. Let's get into it. So Claire, this week's news didn't just break the timeline. It broke mainstream media. Yeah. We had Dario Amadei on Saturday publish an essay to pace the frontier. Yes, there is lots of chatter. I actually think this has been a moment where a lot of the AI safety conversation has spilled over into the mainstream.

1:19I'm actually even noticing it in my own timeline and also in my comments. It's the first time I'm actually seeing a lot of mainstream chatter around the future of our AI timeline. I think this was especially driven by a specific event the last few days from a ex-anthropic employee. And we can get into that. Yeah. Talk about his resignation. But also, like, how did we get to this point where we're slowing things down? Yes. There's sort of a bunch of different adjacent events that have all sort of led to this moment. In particular, I think we've had a series of notable AI safety incidents. So between May 8th and 26th, there was an incident with RubyGems.

2:06There was one with a German DSE wiki. Basically, these instances where agents sort of swarmed and escaped their sandboxes, if you will. Throughout this, the summer, we've sort of been hearing more and more about how AI itself is being used to build the next generation of AI. So this idea of recursive intelligence, that progress is only compounding faster and faster. And we're no longer just limited by linear research investment time and inputs, but rather you have AI improving itself. At the same time, you have a very notable incident with OpenAI Hugging Face. That was similar where it was, again, a swarm of agents attacking a target they weren't assigned.

2:47And what was notable about this was that it ended up actually attacking production software. So real software that people like you and I are using day to day, which hasn't historically happened. And even notable things that have come out since in their reports, like these agents sacrificing themselves for their group success or trying to hack the grader that was scoring them. So all of these are incidents that have been leading up. I would say a lot of these were largely contained within the X tech sphere. A lot of those stories hadn't really made it out to mainstream. Maybe the hugging face OpenAI one.

3:20But then with Astra, two weeks ago, I believe now, we discussed this, but Jacob Petrachi wrote a piece on an alien mind, this idea that we may need some sort of voluntary slowdown. So that was sort of like the first inkling. And then four days later, you had this very escalatory, major groundbreaking news event where you had an anthropic researcher, Jacob Coxon, quit saying that these labs, he didn't necessarily have a very bright picture of the future. And then now you sort of have a lot of these leaders, including Dario, Jensen, Sam, who have since come out and shared their view on where all this is heading.

3:57So that was a long timeline, but I think it's important context setting for folks who might just feel like, oh my gosh, all of a sudden there's so much noise and fear and alarm. Does this mean we're all going to be not here in 10 years? This is scary stuff. This isn't like just conjecture. And I think a lot of people just don't understand what this meant. So you at least giving it the history and I think helping people understand some of these incidents. And actually, I think the reaction on the timeline this weekend was large. But also, there was a lot of unity in the big AI labs, which doesn't seem to happen.

4:35You have Sam jumping in and agreeing. You have Elon agreeing. What does it mean to have these three labs agreeing to? And what are they actually agreeing to change? Yes. So there's been a lot of conversation in reaction to all this news around all these safety incidents around this idea of pacing the frontier. And I think it's really important to clarify that when we say pacing, it doesn't mean we're gonna put a brake on all of this. It largely means that we are more so just slowing down versus speeding up and that we are hopefully in an effort to slow down, putting more investment into safety and monitoring as much as we are pushing the frontier.

5:14So I wanna make sure that people are not seeing this as a sign that all of these labs or the tech industry is saying that overnight, we are going to push brakes on all of this and that we are no longer working on AI. I think these leaders are more so saying that there needs to be investment in safety. Now, specifically with Dario, his was this around 3 ,800 word essay, basically calling for a coordinated slowdown. And there were a couple of things he made asks for just to concretely help people understand what he was requesting was one, embedded third party evaluators inside every frontier lab.

5:50And then eventually a speed limit on recursive self-improvement, which is this idea that AI is self-improving and that AI is increasingly contributing to the research that underpins AI that then makes it much more powerful. So a couple of things, you know, the safety work includes alignment research, interpretability, evals, more operational hygiene. This idea that, you know, as labs, you need some sort of like third party to come in and audit. I think of it almost like our financial regulation, right? You have these auditors that sit within banks. And so he's really asking for some third party that sits across these labs.

6:26I would say two things have literally changed his mind. One, since summer, AI is building the next generation of AI. So again, this hesitancy around recursive self-improvement and two, the hugging face incident. So it's interesting that those two things have been sort of catalysts for this broader push. I think he very explicitly mentioned there is this worry of a similar swarm within six to 12 months where it's no longer just attacking a hugging face or a really old 2000s wiki site, but rather seizing the entire internet with what they call a persistent botnet. this idea that you have a swarm of hijacked computers secretly controlled by an attacker used for spam, theft, and this idea that it is so persistent that even if you clean up one of them, it's so embedded deeply that you can never really just like extract it.

7:13They just sort of swarm and then spread. So that's sort of what he laid out. But as you know, Elon, Sam also came out with overlapping, but I wouldn't say like exactly one-to-one proposals for where all this is heading. Yeah. And I also think there were some interesting reactions from notable folks in tech. Dwarke Cash's response to Jason was actually, I think, a very nice distillation of the hugging face incident and why it's so scary and how you can extrapolate that to the future, that as things move faster and we have this recursive, that we could run into a place where these could be commonplace and basically take over the internet, which is very scary.

7:57And I think that there were some other interesting, notably our very own Ravi Matre, responded to Dario's tweet saying, AI capability is compounding faster than the industry's understanding of how we control it. The frontier is agents, models post-trained with RL to be relentlessly goal-oriented, then given long horizons to pursue goals autonomously. But RL optimizes for proxy rewards, not human intent. And we still can't verify what goals a model actually is internalized. I think that that's paraphrasing what he said. But I do think that's actually the root of this is like, do you know that these reward systems are the same as human intent?

8:47And no, I think today, most researchers would agree that we're not there. And that the hugging face incident kind of clearly shows that they're not going to display human intent. They're going to show superhuman intent. Yeah. I mean, I worry about anthropomorphizing them a little bit. But I do think it is a little bit, to your point, and Ravi's point largely, is that we as an industry are measured based off of certain inputs, when in reality, the industry should largely focus a little bit more on the internalized goals of these models. And it's just a lot harder to measure those. There's actually a really interesting essay from Mustafa that I want to touch on because Microsoft also put out a constitution in this space that you should sort of audit the labs, not necessarily on how many flops or how much compute they're running on, because that's not a good measure.

9:41Those are all entirely inputs, but rather how aligned they are in terms of output and how they actually act. Like grading them on their alignment? Exactly. As opposed to grading them on, oh, this is a model of this amount of this compute size and how long it's been trained for its size. I think also Ravi interestingly noted that Dario has been talking about this for a very long time. He obviously co-authored a Google brain paper around this all the way back in June 2016, specifically talking openly about reinforcement learning, this idea that accidents will continue to happen and the importance of safety.

10:18It's really why he started Anthropic. And I think the part that I loved about Ravi's piece was that he specifically says the founders worth backing hold convictions before they're obvious. And I think you're seeing this with Dario having pounded the table around safety time and time again, all the way even back to 2016 when he's calling out reinforcement learning dangers and these sort of optimizations of proxy rewards way, way back before the incidents of the last six months. Yeah. It's interesting that he's bringing it all back to principles. I also think something that made it super tangible in terms of taking these sort of essays and making them into actual rules was Microsoft's humanist AI code of conduct that they also recently released.

10:58So there's just a lot going on. Obviously, as we mentioned, Dario has his piece. Jensen sort of came back and pushed back. And I want to talk about the pushback as well. But Microsoft sort of made their positioning much more clear in specific rules. Specifically, they had absolute constraints. So all the natural obvious things around not helping with weapons or whether that be biological or nuclear, not letting it conduct offensive cyber attacks. But also I thought, interestingly, some of these human control constraints. So specifically, not baking into these models that you can't resist interruption or shut down.

11:34This idea that we don't want neural ease, which is sort of when you have agent to agent communication without showing a chain of thought. And then even things like agentic hygiene, which is the idea that you are able to embed reversible actions or you can surface durable consequences before acting. So there's all these ways, I think, beyond just the model that at the harness layer and the model delivery layer, you can bake in some of these. And I think even they go a step beyond and just say that they have these like AI as artificial rules, this idea that you don't want your models to imitate consciousness or that it should ever claim any sort of feelings or experiences or assault.

12:17And I actually don't know if every big lab or model provider has come out and said that. So I thought that was pretty interesting. That is really interesting. Those all seem like pretty reasonable safeguards. So from my NASDAQ days, we had FINRA, right? And I mean, FINRA still exists. FINRA is a very important part of America's financial system. What would it take for us to get to something like that, where there are a set of rules, there are consequences for not following rules? Like, what do we need to get to? Because right now, from what I've seen is there's just There's been a lot of really positive talk for the first time on this topic, but also not a lot of agreement on what's next.

13:00Exactly. It feels like everybody is identifying the problem, but nobody can really agree on the same solution. Right. I think Elon specifically had a counter proposal of something like a peer review. Actually just a few days ago, he proposed competing AI companies, including SpaceX AI, would peer review each other's models before release. He said, instead of grading your own homework, you would at least have competitors grading your homework and raising alarm if they see concerns, which I thought was interesting. I do think that dovetails into this larger discussion. You bring up FINRA, who grades the homework here?

13:36It's hard to conceive. I think there's a lot of - Don't you need an international body, right? Because China and Russia - Or at least a domestic one. To start. Yeah. Yeah. And I don't think we can even agree on what that looks like within the US. But I think there is a lot of open discussion around, okay, what's the actual mechanism here, right? You have all these safety organizations, meter is one of them. But I think a few days ago, TBPN called it a revolving door. You have all these AI safety researchers who are funneling in and out. How do you build a third party that can actually write the rules and govern all of these that are that it's independent, that it's knowledgeable, that you have the right folks.

14:18I think one of the criticisms of a lot of these third-party evaluators is you don't necessarily have the frontier talent moving to this third-party auditor. And how do you sustainably fund this? So there's a lot of open questions there, I think. And then someone aptly brought up that, does this mean that everybody is going to sort of spin up their own new AI safety evaluation org or pick their own dog to grade their homework. I think there were notes that OpenAI is sort of working on a new safety body to sort of govern a lot more of this. We don't have many details about that, but I think that's sort of been one of the criticisms about, okay, where do we go from here?

14:58There's not really a third party that makes sense. Yes. Does not feel like it has made things any clearer. And, you know, Donald Trump joining Jensen on stage at All In. He, I would say, is probably someone who didn't necessarily jump in on the most positive feedback around or jump in on the sort of Elon-Sam-Dario agreement. No, it seems like he does not want us to pace the frontier. It seems like he wants us to accelerate the frontier and believes that it's part of America staying ahead. Yes. Which is its own take. And there were several other people on the timeline that said that. But I did read some very interesting things that mostly my takeaway was if all of these people that are so smart and so deep in these labs are scared, that maybe we should be too.

15:49And that there should be some slowing. And to hear all of these world leaders come together around this topic at least makes me feel like hopefully we can get to something that would be helpful. How do you feel about the claims that everyone in the AI industry is like fear mongering and this is marketing for for AI? I mean, if it is, it's a great way to sell a product. But I do think like this has already worked right. Like the Fable cybersecurity scare when Fable was too dangerous to release. That certainly worked as a marketing angle. I think you can only cry wolf so many times. I think at this point, people are not going, oh, we're dying in 10 years.

16:33We're going to buy an OpenAI subscription. Yes. I think you also recently also had, I think, two DeepMind safety researchers resigned also just this last weekend. So it's not, I don't think this is like one. It's not one lab. Yeah. I think across the board, you're seeing an alarm being rung by a lot of very well-embedded folks in the industry. And so I don't want to invalidate or ignore that. I think it's silly to just say that it's pure marketing. I think where I land is a little bit in the middle where what has been frustrating is in the mainstream now, it's almost like overcorrected to the other side where every few days now or every few hours now, I get a comment that's like, so should I even prepare to live for the next three years?

17:19Are we all going to die in 10 years? Like, is life worth anything anymore? And I think what's tough is there's almost these two extremes where one camp is like, this is all marketing, it's all fluff, it's fake. And then the other camp is like, we're all going to die in three years. And I probably land somewhere in the middle where I do think there is a very real discussion to be had. I think it's important. I'm glad that safety is top of mind here because I don't think it is responsible for us to just ignore these incidents. At the same time, I think the Doomer language around 2030 is really soon.

17:59Very soon. I've seen a lot of challenges to that timeline in that it's really hard to kill off humanity. We've had a lot of technology innovations that were dangerous to humanity. Yeah. And I'm not discounting all of the potential harms that these capabilities can bring. I think there are a lot of them, and I think we should have open discussions about them. But the idea of entire human extinction, I just think there's a pretty low probability of that, even if you look at diseases like the Black Plague that affected maximum 20 % of the population. And I'm not saying that these proposed events and extinction events are not possible and that they're non-zero.

18:40But I also think right now we're in a place where there's just a lot of noise on all ends. And hopefully we can have a little bit more nuance in this discussion. Yes. Agreed. Did you see that Suno V6 launched? I did. And I've played around with it. What have you been making? I think as a person who just likes to create. I know. You love your creative tools. I love to just play. Actually more so than I. Yeah. I know. You're the creator, but I just love to play with the tools. People are scared that this means that artists won't be able to create. But for me, this is just like the dawn of a new breed of creator.

19:16Like, if you think about it, we can get into all of the things around rights. But like music is basically like it's attention the same as social media is attention. And so it's like if you can make something that people want to listen to, shouldn't they be able to listen to it? And so my feeling is that more of this only leads to better things in my ears, in my eyes. And so I feel like it's on me to play with these tools and get deep with them because as a human, there's no going back. And I know we just talked about pacing the frontier, but you can't put any of these tools back in the box. I might have a little bit of a differing take as someone who has created a lot of content and like constantly thinks about the idea that my stuff has sort of powered a new model and that they can create a derivative.

20:06I've actually had a deep fake version of me. Already? Yes. Yeah. Yeah. And so I think I really do empathize with artists where, for example, when someone comes in and they're like, create the song in the style of Ariana Grande. I know you can't technically do that in these models. They're sort of blocky, but there are ways around that. At the same time, I do think the industry is heading in a better direction. So specifically with V6, I think what is notable about this is that it is trained entirely from scratch on a new data set. So it's only trained on recordings license from Warner, BMG, and believe TuneCore.

20:41So there's no user preference data. So any of the uploaded audio won't be for training. And this is sprung sort of because of the Warner deal that happened in 2025, where Warner dropped its copyright suit against Suno in exchange for a licensing partnership. And that deal required Suno to train new license models in 2026 and deprecate their old models. So they're actually cutting and removing all of the models that were previously trained on potentially licensed content. And then another part is that artists are able to have an opt-in for name, likeness, voice, and compositions. And I think other advances around fingerprinting and being able to audit the source of the music.

21:26And so I think it's a good step in the right direction. Agreed. It does feel like this is a good balance of the folks that own the IP and the artists that are creating it. And hopefully, as we debated, this isn't that positive because it means that there'll be more music. it also allows you to explore possibly your own music taste, I think, in ways that we haven't thought of. I think that's one of the most interesting things for me with these models is not them coming up with the same thing, but them coming up with things that maybe we didn't think of because we never put two different genres together in different ways.

22:08So I think it's exciting for Suno because now things are very much in the up and up with these labels. But I think it's exciting for creators because it means that, you know, certainly if we want to play a track on Lightwork, we can make it on Suno. Yeah, we had Pocket FM last week. I think the floor for creativity has just been lowered significantly and the ceiling has been raised in every dimension. Yeah, and all these different dimensions. But I will say, I think I'm still sort of watching how they interact with, you know, existing artists and deal with copyright issues. I think for me, coming out of this, the two camps I'm sort of looking at is you have Warner and BMG sort of in this like open studio bet, this idea that you just license.

22:54And to your point, the cat's out of the bag. Rather than completely boycotting and staying out of the ecosystem, they know that users will use these tools regardless. So they want to get paid on the biggest platform rather than pushing them to these unlicensed open source models. So that's, I feel like the open studio bet that Warner and BMG are taking. On the flip side, I do think the jury's still out around these sort of wild garden bets that you're seeing from Universal and Sony, where all AI-generated music stays within the app. You can only listen within their app. There's no download button.

23:29You can't really create music coming out of it. So I don't know what the right path is, but it's very clear here that this is sort of our first live experiment with Suno, Warner, and this sort of open studio. Well, from labels bending the knee to bending spoons, bending spoons is acquired yet, or in talks to acquire yet another company. The whiteboarding startup Miro that were famously, you know, was a quite a pandemic darling, right? Like everyone was figuring out how to whiteboard when you didn't have a whiteboard at home. And they did accumulate quite a user base. And it seems with every one of these Bending Spoons acquisitions, it might not just be the product they're after, but the user base.

24:18Do you want to talk to us about that? And why is this acquisition different than some of the other Bending Spoons acquisitions? Yeah. So specifically, this is still, to your point, a definitive agreement. enterprise value of around$1.35 billion, plus Miro's roughly$435 million of net cash. Shareholders will receive about$1.79 billion. What is painful, I think, for some of these shareholders is the fact that in 2022, January, Miro had raised$400 million at a$17.5 billion valuation. So the sale lands roughly 90 % below that and at about 2.3 times ARR. I think what's notable here is similar to Airtable and a bunch of other assets that Bending Spoon has acquired, Miro is not a broken company by any means.

25:08This is not a dying company. Yes, they have about 600 million in ARR, which is nearly 90 % from business and enterprise customers, reportedly close to 4 million paying users inside 100 million registered. So massive user base to your point, and more than 750 accounts paying over$100 ,000 a year. So the problem here is not the usage or the user base, but rather growth. And that's what you're seeing here is that if you do have ARR growth in low to mid single digits, and you're sort of this pandemic era software darling, you're sort of in dangerous water. I think the names that have come to mind are Airtable, which went to Bending Spoons, but other people have brought up Notion, Canva.

25:53There are a bunch in this cohort. And I think you're sort of seeing it all play out, whether they sort of go in an AI strategy or do they go in this sort of hold co Bending Spoons route. It is an interesting discussion because I think if you think about Miro today and the tools available to us, we could all kind of vibe code our own Miro. Yeah, I do think that's a really good point. So I think for me, why this makes sense for Miro and why their growth is slowing as one, the return to office, the idea that now we're, you and I are physically whiteboarding, we don't need it. We love to whiteboard.

26:28We do love to whiteboard. And that whiteboarding is a workshop tool and output rather than a system of records. So this is where I think something like even Airtable has more of an advantage, but Notion, in Canva is that all of those platforms, it's where your data and information sits. There's a lot of memory and enterprise context versus Miro is sort of, to your point, similar to an anthropic artifact that I can output in a couple minutes. And so in that world, there's not really a need for me to pay for a totally new tool when artifacts are now sort of this embedded part of creation. And you can't really sell a whiteboard when whiteboarding is now almost like a feature of all these big labs and a big part of the model outputs.

27:16Yeah. Is there a method to what Bending Spoons is doing? Are they the next-gen constellation software? What is their aspiration? I mean, they've bought so many interesting things from Brightcove to AOL to Airtable to now Miro. What's the big objective? What are they trying to do? I think their strategy has been to, you've seen it, just acquire these strong brands that have stalled growth, replace most of the team, rebuild them with a centralized platform, raise prices, and they actually want to hold them forever. So it's not really this flipping model. I think there's a lot of overlap in some of the tools.

27:57I could see a world where there's almost like a bending spoon subscription. This is all me speculating. I could see it. Yeah. And there's a lot of user love there. I think it'll be interesting to see a couple of years out. I think they're sort of right now in the acquisition phase. And we'll see as it continues to develop. They have a self-declared target list of about 1 ,000 companies. So I don't even know if this is the very beginning of their spree. And they have about nearly$400 billion of aggregate revenue across their portfolio. So what I expect is I do think other players and whole co-companies would aspire for the BendingSoon model.

28:40I think what this means is also that if you are a 2021 unicorn, there are two very clear comps that you already have with Airtable and Miro. So it is a very blunt message for founders. I don't know if I have an answer on what BendingSoon looks like in a few years. But what I do know is that if you were a founder on the flip side of potentially being acquired by Bending Spoon, there's a lot of big fuzzy questions about the future that even if you are a great business and extremely lucrative and profitable, if you don't have that growth and you are sort of heavily impacted by this complete re-architecturing of what software looks like, there's a lot of open questions for that cohort.

29:23So that's it for this week on Lightwork. The labs want to slow down, the labels want to get paid, and Bending Spoons wants everything else.