Does Europe have a voice in AI safety?

0:02Hello, and welcome to the Sifted podcast. I'm your host Freya Prattie, Sifted's Associate Editor. Joining me today is senior reporter Daphne Le Prince-Range. This is your second pod in two weeks and you're joining from your new flat. I am in Paris, yes. Nice, I'm in my flat in London. So we're going to be talking about AI safety. Last week, the resignation of anthropic researcher Jacob Coxon catapulted the discussion over AI safety into the mainstream. A thread he posted on X quickly garnered millions of views. Coxson later gave an interview to Wired where he said, the consensus is that the next year or two is crunch time for humanity.

0:42He said that was a literal quote from colleagues at Anthropic who regularly use the words end game or crunch time. He said, from their perspective, this is when Anthropic and its competitors decide the fate of humanity. And we also have some people in Europe making similar statements, I think, Daphne. Yeah, definitely. I think the reaction in Europe to what has been going on last week is probably double-sided. I've definitely heard some skeptics. For a bit of context, obviously Anthropik and OpenAI are both preparing for an IPO. And you could claim that they can benefit a lot from saying that the technology is the most powerful thing humankind has ever built.

1:21and what I find interesting with that criticism is that often what comes from it is that this narrative unfortunately displaces the focus from the real risks of AI which are maybe not so much civilizational but more around job losses or how using chatbots affects our critical thinking or how these systems have huge energy requirements that are contrary to many of current climate goals so that's for the skeptics and then there's obviously you know also quite a lot of people who agree that there are massive and increasing threats from AI as you mentioned we've had posts this morning you were mentioning the post we had this morning from a former London-based DeepMind engineer who was saying he earnestly believes that AI has the potential to kill us all I was speaking to a security engineer yesterday as well who was using the word horrifying quite a lot and to describe how he felt about the pace that AI is developing at I think there's particularly of a lot of fear around the cyber risks of AI and how AI agents could potentially hack into very well protected systems government systems and then all the all the risks that this could generate a turning point probably came actually not necessarily last week but I'd say There was a turning point in July when OpenAI confirmed that one of its models accidentally hacked the systems of Hugging Face, which is a France and US-based AI company, which was recently acquired by NVIDIA.

3:05So this was one of the first major cyber attacks that was perpetrated by AI systems autonomously. And we saw AI systems organizing to hack into a system. And I think that was a real wake up call for many in the industry. um in europe i would say there's also a feeling of slight powerlessness perhaps uh in the sense that we don't have ai labs of the scale of those in the us who are building at the frontier of ai who are building these technologies that have potentially existential implications for the human race if we are to follow the narrative um that that has been overwhelmingly taking over the past few days.

3:48So these are mostly in the US. We've obviously got quite a lot going on in China as well, which is not very far behind the US and emerging as a key player in the AI race. In Europe, we have Mistral, which is based in France, which launched to build AI models, but increasingly the consensus is that they are not building at the frontier anymore. So I think there is a real question of how much of a voice does Europe have in this debate and in a way that that power doesn't seem to be really in the hands of Europe. How do you think, given all that, companies here are approaching safety then? In Europe, companies are building more on the application layer, what's known as the application layer, which means that they leverage the models developed by leading AI labs to create applications for customers.

4:36So obviously the understanding of safety and security is slightly different there. but I wouldn't say that European companies are not thinking about this at all it's quite the contrary these companies still need to ensure that the products they're building and selling are reliable and trustworthy for their clients so it'll be stuff like making sure say you're a say you're a company that you know provides an app to create videos thanks to AI you don't want that to be leveraged by users to create deep fakes or to spread misinformation. Synthesia, which is a video generation company based in London, their technology, for example, was used a few years ago to create fake news videos that were promoted by the authoritarian government in Venezuela.

5:24So there's definitely concerns around safety and security. They're just slightly different to those that have been voiced in the past few days by the leaders of the big AI labs. But I definitely see this becoming more and more of a concern in Europe. Do you think building safe AI is or will become part of Europe's edge with customers, with potentially government contracts, I guess? I think definitely. This is somewhere where Europe, and we might talk about regulation at a later stage, Europe was quite a bit of a precursor for the best and for the worst in in terms of defending this idea of a regulated, trustworthy AI when it started working on the AI Act.

6:07Europe started working on the AI Act in 2021 and the regulation came into force in 2024. So that was years ago. And at the time people criticized it, but I think increasingly the times have changed and now building trustworthy AI is becoming an edge. And obviously to us where the AI can mean many different things. It can mean AI that is transparent, AI that is reliable, AI that won't shut down if geopolitical events make it that access to some technologies are revoked. So it's a big term, but I think generally we can describe it as AI that carries out the tasks that the user intends them to and will not misbehave.

6:52And I do think increasingly this is where Europe is potentially finding its edge. A company like Mistral is definitely a huge part of their pitch. Mistral recently raised a massive Series D. They raised 3 billion euros last week. We were talking about it on the podcast. And if you read their press release, so much of it is around these notions of trust and control over the AI. So yeah, I'd say this is where a lot of European companies are positioning themselves now. Yeah, that'll be interesting the question of whether customers will go for a company that is safer but perhaps not better in terms of capability or product when their competitors are maybe foregoing the emphasis on safety.

7:33I don't think we've, there's for a long time now been a lot of question about sovereignty and pursuing sovereign solutions and I don't think we're seeing that many customers pick them just because they're sovereign they also have to be better so I guess it's going to be the same question with safety right if competitors are using something where the company hasn't put as much effort into safety but it's way more capable picking a slower option is going to be very hard for customers is is my read of it yeah definitely I think there is that duality between safety versus capability the engineer I mentioned I was speaking to yesterday was telling me quite rightly the the most productive efficient AI system is a system that is completely unconstrained obviously um so there is a toggle there but as as an enterprise that is deploying ai you also don't want an ai that's going to hallucinate or for example produce deepfakes or spread misinformation or you know provide financial advice when it's not intended to provide financial advice so as an enterprise that's deploying ai you also don't want an ai that's going to misbehave so i think that will be kind of a balancing act there that's true i had a really interesting chat with a guy who's left DeepMind recently to start his own venture he called it he wouldn't say whether it's a company or something academic but he wants to build better reasoning into these models because he says that will both improve the safety because you can see how it's got to its conclusion but also make it more effective so if we could yeah as you said tie the two qualities together it would it would be extremely beneficial I think.

9:08To dig in more on the EU AI Act which you've already mentioned do you think this moment is then kind of vindication for those policymakers that pursued that five years ago now or does it show that their efforts will be meaningless because this technology will just be so so powerful in the face of regulation it's it's a tricky one it's probably too early to tell because the euai act it was it came into effect in 2024 but it's becoming effective in steps so it didn't immediately come into effect but i do find it interesting how a few years ago, many were criticizing Europe for regulating AI and now AI leaders in the US are calling for more oversight and under slowdown.

9:49In terms of the act itself, to answer your question directly, I do think it's already behind. The AI act's regulated for high risk models in 2024. It doesn't include anything about agents, for example. So it's just an example of how quickly the technology is moving and the fact that existing regulation is behind. But we have seen it's had an impact, I'd say. There's been regulation around AI emerging in other parts of the world with a similar model or pitch off the top of my mind in California. In the US, there is regulation around AI. So I think Europe will probably continue to try to establish itself as the third party regulator.

10:30Whether it's successful or not, obviously remains to be seen. Europe is, even though it doesn't have massive AI labs, It is a huge market. So what it does is not completely insignificant. But I think one of the important things here is that any regulation would need to be very much intergovernmental. So if Europe is just regulating on its own, it won't really have any effects. If the US is deciding to regulate or slow down on its own, that is potentially too risky because it means that other players like China might race ahead and develop more capabilities. to your previous point around safety and necessarily restraining the capabilities of these models.

11:07So I think it's fair to compare this moment in time to things like climate change or the nuclear arms race where, you know, regulations might end up looking like AI companies that are building with a certain level of risk should be spending X percent of their revenue on safety and security. But this needs to be across the world in, you know, in agreement with all governments. I'm not sure we've had a working model for such regulations to come about so we'll have to see what it actually looks like yeah the idea that a hostile actor would think we won't attack this French company there under the AI act is laughable one of our colleagues has worked on a piece about AI agents as you mentioned and the their interaction with regulation in the AI act so we can drop a link to that piece within the episode notes as well for our listeners do you think one of the effects that we might expect to see would be money now pouring into companies working on the defense against this hostile AI threats for companies governments are we seeing that in Europe yeah I say we are seeing that and particularly again on the cyber security side of things the financial loss from not being cyber secure is is extreme Whereas with other things, that kind of financial impact might not be as obvious immediately.

12:29So we're seeing a lot of companies building in the cybersecurity space to help protect against AI-powered attacks, which are getting ever more sophisticated by the day. So lots of money pouring into that. But we're also seeing different types of companies as well. I'm thinking of companies that will build kind of a sort of layer that sits between the users and the models to kind of continuously test behavior and enforce policies to avoid things that I mentioned before, like hallucinations or unsafe agent actions. another type of company i'm seeing is around uh what companies building around transparency so enabling you know more insight into where the agents are why they are there how they make their decision what kind of data they're using so increasing the explainability of ai um so yeah we I think we are definitely seeing a wave around that.

13:33One thing we are noticing, it's been the case with AI since the start, but a bit of a talent crunch, I'd say, with many of these qualified experts often going to the US because that's where the AI labs are and therefore that's where they want to be working, nearest to the technology that they're building, safety and alignment around. So that will be also a challenge going forward in Europe, I think. Well, now I guess we have a number of talented researchers very publicly leaving these frontier companies. Perhaps they'll join companies working on safety. That could be a good talent move, perhaps.

14:10This podcast is brought to you by Stripe Billing. Here's a problem most AI and SaaS founders run into. Your pricing needs to change and suddenly it's a multi-week engineering project. Stripe Billing gets you out of that loop. launch fast, test pricing models and optimize revenue without touching your backlog. Whether you're selling subscriptions or seats, Stripe Billing lets you adapt pricing as you grow. And when you're charging for tokens or other usage, Metronome, a Stripe product, gives you real-time metering, rating and billing across the dimensions that matter. Learn more at stripe.com slash billing.

14:45S-T-R-I-P-E dot com slash billing. One group I want to talk about in relation to safety, we had some chats about this earlier. VCs are, I mean, Anthropic, OpenAI, they're on those cap tables. This is the kind of capital pool that gets these companies to the point of being able to push at the frontier of technology. VCs are super impactful within this question. I said to you earlier that I think last week was the first time a VC had brought up safety to me in one of my coffees. Is that the same for you? are VCs talking to you about safety? Not so much to be honest I think it's moving slowly perhaps it's definitely not something like you know ESG which is something that a lot of VCs would have been talking about or pitching as a criteria that they are looking at when they're investing and perhaps AI safety will eventually be included in that it would make sense but no to be entirely honest I'd say it's on the side of VCs I think it's still and as you know it that is their job it's still led by the possibilities of profit and making a return on their investments so I wouldn't say it's a priority yet maybe things will change after this week though.

16:01Yeah and I do think since looking into this a bit more as of last week in terms of the due diligence VCs are doing there's a firm called Radical Ventures which has backed a few of the new labs who seem to have very comprehensive methodology for thinking about safety. General Catalyst people have said to me have been doing good thinking about this. I'm going to talk to them later. So that will be in one of our articles this week. So I do, there seem to be pockets of people thinking about it, but I, yeah, as I said, I really noticed how few people actively bring it up with us. It is worrying, I think.

16:35Daphne, what else are you working on? What can we expect from you next so one thing we have been wondering this week with all this talk around AI safety is how things might change in Europe and particularly with the right the rise of AI labs so I obviously mentioned first that we do not have in Europe AI labs of the scale and size of anthropic and open AI but this year we've seen quite a few labs launching which are obviously all very early stage, building at the frontier of AI with the idea of reaching super intelligence. So it's obviously still too soon to know where that will go. Some of them like Inherent Labs in London and Ineffable Intelligence.

17:20They've been set up as a public benefit corporation, which means they remain a for-profit corporate structure, but their financial goals are aligned with a specific social mission um so that's really interesting i don't think we'd seen that in ai in europe before uh we'll definitely be be tracking where that goes um so inherent labs specifically our colleague is writing a a profile um this week that i highly recommend you read uh which will have provide a bit more detail into that specific structure um but yes generally i think we're interested in finding out more about these governance structures and whether they will become more of a thing going forward especially in Europe where these AI labs seem to be multiplying now.

18:06I've heard a few things that are going into the inherent profile from Maya it's going to be great so look out for that one it'll be a great read. Great thank you for joining us Daphne. Thank you. Listeners if you want to follow our coverage of AI sign up to Daphne's newsletter which goes out every Monday, as well as our daily newsletter. We'll drop a link to both in the episode description, as well as our coverage of AI safety from last week or so. As always, listeners, please rate, review and share the podcast. And this episode was produced by Maya Duran for Hornby.