#129 Alexandra Geese: Demystifying AI Regulations in Europe & Beyond

0:00Artificial intelligence is a set of technologies developing at astonishing pace and then it's very difficult to say at one point in time there is this one piece of regulation and that's it. We do need artificial intelligence that serves humanity, that serves the people. And it's our task as elected legislators to make sure that that happens. The Council came up with its position, which is called the general approach, earlier than Parliament. and something happened in the meantime before parliament approved its position and that was called chat gpt and that was the moment like a week or two after chat gpt was rolled out in europe and everybody was using it when the majority in the european parliament completely changed Hi, I'm Craig Smith, and this is Eye on AI.

0:58This week, I talked to Alexandra Gies, a member of the European Parliament. She's worked on the Digital Services Act in the EU and is now one of the members drafting the AI Act. Parliament has just passed an initial draft of the Act, and it's now under negotiation with the European Council and the European Commission. This is the first significant, comprehensive attempt to legislate AI, and everyone in the world is watching what the EU is doing. Certainly the United States is expected to follow a lot of what the Europeans come up with. And OpenAI, Microsoft, Amazon, all the big AI players, Alphabet, of course, have been lobbying the EU to try and influence the act, try and soften the act.

2:04So I wanted to hear from Alexandra how the process has gone. And I think anyone interested in AI legislation will want to hear what she has to say. Before we begin, I want to give a shout out to another podcast. This one's by TechCrunch. It's called Found. Each week, TechCrunch Plus reporters Becca Skutak and Dom Madori-Davis. talk with a founder about what it's really like to build and run a company from ideation to launch. I'm interested and listen to the podcast because I have my own project. They talk to founders across many industries and their conversations often lead back to AI as many startups start to implement AI in what they do.

2:56New episodes of Found are published every Tuesday, and you can find them wherever you listen to podcasts. So I encourage you to do so. Now, here's Alexandra. I'm a member of the European Parliament. I'm from Germany, and I'm with the Greens EFA Group. So that's the Green Group in Parliament, and I'm a member of the Green Party in Germany, which is currently in government as well. and I've been working on digital politics since I started in 2019. I was what we call a shadow rapporteur, so the person who for my political group worked on the Digital Services Act, which is a major piece of legislation for digital platforms.

3:47And I have always been interested in artificial intelligence, and they have been following the artificial, the AI act very closely. Okay. So the AI Act, the first draft has been passed by the parliament. Now it's in a three-party sort of negotiation, is that right, to nail down a final version? uh i guess before we begin can you give the the headlines the the sort of top uh line uh explanation of of the acts of various clauses or the main points of the act i know i mean i know what they are but for listeners that haven't been following it well yes it's um the act is trying to be a very dynamic piece of legislation because we are quite aware of the fact that artificial intelligence is a set of technologies developing at astonishing pace and then it's very difficult to say at one point in time there is this one piece of regulation and that's it So what we're trying to do is to introduce different categories and to look at applications of artificial intelligence rather than at the technology itself.

5:23and the categories are basically um bands applications which are banned which is for example um social scoring in chinese style that's that's one of the most important bands another ban is the one on remote biometric identifications in the public sphere therefore cameras in the public sphere that in real time can tell the authorities where you are, where you have been, what you have been doing. This is something we know very well from the Hong Kong protests where people could only protest completely masked and look very fearfully at these huge cameras installed everywhere and this is a situation we would like to avoid in Europe.

6:10So those are examples the parliament introduced further brands for example on applications that claim to be able to to recognize your sexual orientation just to give an example then there's a second category which is called high risk which are a long series of combinations of applications in different fields like recruiting, for example, border management, education, where applications need to go through an authorization or licensing procedure. So companies producing them have to decide whether their tool, their AI tool, fits into that category and then go through this procedure and need to be to get an authorization by a national authority in one of the country, the member states of the European Union.

7:10They also have to have documentation for what they have been doing, what kind of data they have been using for the modeling, so on. And they need to make sure that at every time of the process, there can be a human person, a human in charge. And then there is basically the biggest category, which is low risk, which just has just a few documentation obligations, but doesn't need to be authorized, but can be marketed right away. The parliament introduced the fourth category, which are foundation models. So basically large generative models, which has become famous with ChatGPT. And therefore we foresee a few obligations also for producers of these large foundation models.

7:59Yeah, actually, can we talk about that? Because that's what initially spurred my desire to talk to you were comments later retracted by Sam Altman saying that open AI may have to withdraw from Europe. I mean, he later said that they were not going to withdraw. But can you talk about that fourth category and what obligations it puts on people that build these foundational models? Well, providers of foundation models have to, and I'm going to read this out to get it right because it's long, I want to demonstrate through appropriate design, testing and analysis, the identification, reduction and mitigation of reasonable foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law.

8:57prior and throughout development with appropriate methods such as with the involvement of independent experts as well as the documentation of remaining non-mitigable risks after development which i think is interesting because it sort of breaks up this narrative well we just provide the the technology and then we don't really know what happens which you know for us europeans is a little bit shocking coming from the us that that people provide this enormously interesting and powerful technology and then the developers themselves say well we can't really control it we don't really know what's going to happen you know in every other industry sector you would expect a provider of a product to have a vague kind of idea what that product could do in terms of harm and i understand that for artificial intelligence this is quite difficult but at least um we we require them to make an effort to to figure this out and i think another important obligation is is the one on on data governance so um it says they have to process and incorporate only data sets that are subject to appropriate data governance measures for foundation models and particular measures to examine the suitability of the data sources and possible biases and inappropriate mitigation, which is extremely important because one thing we really want to avoid in Europe from the beginning, we're very concerned about was bias, which we know most artificial intelligence entail.

10:30And it's very difficult to put the responsibility on companies using AI tools that are based on foundation models and then tell them, well, well, you need to get rid of the bias. I mean, how are they supposed to do that if it's in the foundation model? So I think it's very important that especially the providers of these big models really reflect on how to avoid bias seriously. Yeah. And the other issue that I've seen raised is that of copyrighted materials being used in the training of these models. um i imagine there's been quite a bit of discussion in in the drawing up of this act uh about the foundational models that are in the market now gpt4 or uh or you know google's bard uh how what do you do now i mean because those models are already trained and they're trained on, you know, virtually the entire English language internet or maybe the internet itself.

11:50I'm not sure how that was segregated. They worked well in German as well, interestingly enough. Yeah. So, I mean, you can't, once the model is trained, you can't take out that information. So does that mean that people providing foundational models will have to train from scratch, start over, and if they want to operate in Europe, provided that the Act passes in its current form? Well, to be very frank, I think it's a question that is currently very difficult to answer. But I think a good start would be oblige the providers of foundational models using that material to be open and frank about it and to be able to distinguish between copyrighted material and other kind of material.

12:51I mean, copyright may be a little bit outdated, but it's still a foundation. the economic basis for human creativity, basically, journalism, arts, that all depends to some extent on copyright legislation. And we need to find a way forward there without just completely abolishing it. I've been talking to artists who are very, very worried about or about what they're going to live on. And I think there's a component really of, that goes beyond just the economic reward or the legislation in terms of what does this kind of technology mean for creativity? Because, you know, this looks, the AI looks very creative, but then its work is based on the work of humans.

13:47And if humans are not rewarded for it anymore in any way at a certain point creativity will end so what does that mean well this i'm not a copyright expert so where this leaves us currently in terms of enforcement of copyright legislation frankly i don't know um the ai act is not legislation yet you explained that correctly it's the parliament's report and now we enter negotiations with the european with the council and then we will have a final legislation and i think in the meantime we will reflect on what that means for copyright legislation i don't see open my eye or bright retraining completely their models but maybe you know looking into paying more attention to what kind of data is used to train these models i think it's it's sort of for um a learning process for us as legislators to understand how artificial intelligence works and what it does to the world but also to to the developers and and the companies providing it to to abide to comply with with existing legislation and to understand needs of society that exact that are expressed by that that legislation as well so i see it as a process frankly yeah um the i mean is there a concern in europe that that if the act is too restrictive uh that that uh europe will fall behind in the implementation of AI and consequently fall behind economically?

15:37There is a concern, definitely. More from the conservative side, I think, than from the more progressive part that I represent. But there is a concern. There is a concern that those products might not be rolled out in Europe, as Sam Altman said. Well, he would just leave. There is a concern of European companies not being able to build on these foundation models and therefore to fall behind. That is true. And I think we do need to take this seriously. On the other hand, the European Union is a huge market. um and the the west the digital world you know it's basically a more western world so the chinese market doesn't count that much because it's in digital terms it's a different universe so europe is still quite important i think for companies like like google and meta and microsoft rolling out ai and therefore i think there there will be some arm wrestling but i think they will listen to us and i know that that many colleagues from the u.s as well um are looking with great interest to what we are doing in europe and hope we might we might be a model and at the same time i think we do need we we do need artificial intelligence that serves humanity that serves the people and it's our our task as elected legislators to make sure that that happens i mean corporations have a different purpose their purpose is to make money for the shareholders so i think we need to tell them what what society wants and i try to listen to to my voters and to the citizens and and figure out how they they want artificial intelligence and what should you be used for And there's quite a lot of fear.

17:38And I think that doesn't help the companies either. People have to have the feeling they understand these instruments, they are in control. And if they want some more control, the providers of AI tools can provide it. And I think that would help us all. That would help the corporations and it would help society. yeah so right now there's a dialogue or a trialogue i guess they're calling it in between the commission the council and the parliament uh to finalize this what are the the does the does the council tend to be more conservative and more pro AI, particularly with regard to the foundational models, than the parliament?

18:33I mean, are we likely to see this draft become more restrictive or less restrictive as it goes through this

18:46negotiation? It depends. It's normally and currently council is a little more conservative than the parliament. And another government might change from progressive to conservative far right in the process in Spain. That's a risk. But there is another, and conservatives or far-right countries tend to be more open to what the corporations want, definitely. This is the dynamics we have observed in Parliament as well. But there's also a second Second issue, which is, or second aspect, which is time, because the Council came up with its position, which is called the General Approach, earlier than Parliament.

19:44And something happened in the meantime before Parliament approved its position, and that was called ChatGPT. and that was the moment like a week or two after chat gpt was rolled out in europe and everybody was using it when the majority in the european parliament completely changed and the conservatives went along with um with a few of the bands um for example the one on recognition of sexual orientation or emotional recognition in education or in water management that kind of thing and went along with the idea to include the foundation models which they didn't want to do before and at that moment i think they realized how how crucial and how influential these models are and that is a good idea for democracy um to at least be able to understand what the risks are and where the data comes from so there is this this aspect of time and of really understanding what a foundation model is and i think this might um entail the fact that council might be a lot more open towards parliament's position as well and the commission also had a very strong learning process the the general approach by council is very close to the original commission proposal and that was also set out before chat GPT.

21:17I think if the commission had to redo it, they would do it differently. And they would definitely include foundation models. Because in the beginning, the idea was just to cover high risk applications. And foundation models at the time were not considered high risk. Now, when you say, well, you know, if OpenAI didn't do a lot of moderation, you could just or asked a chat GPT to explain how to build a bomb or something like that, they realize there is a high risk aspect to it. And I think council will realize that too. So in terms of dynamics in council, let's see. I think I'm very optimistic in terms of keeping the foundation models.

21:59I think the other big issue with council is that the biometric recognition in public spaces, because they are the ministers of the interior that would like to be able to surveil people to a much larger extent than members of parliament think that is useful. So I think that's going to be one of the big issues. In terms of foundation models, I'm quite optimistic. Yeah. On the foundational models, you say you're optimistic or that there won't be major changes required on behalf of the providers of these models. I mean, that they won't have to, that the existing models won't be banned based on the inability to identify copyrighted materials in their training data, for example, and that companies providing those models will have to start from scratch?

23:04You're optimistic in that way, or you're optimistic that those restrictions will be waived for the current generation and it'll be applied going forward? I'm optimistic that the obligations to test and analyze, identify reasonably foreseeable risks and risk mitigation measures will stay in. And the obligations on data governments, that's what I'm optimistic about. What exactly that means for copyright, I can't say at the moment. I think we will need a system to, I don't know, some people have been saying watermark or the copyrighted material or material created by artificial intelligence. Very frankly, I don't think we have the solution yet.

24:04but i'm confident that the oblique the idea of providers of foundation models have obligations for this small for these models and for the use of copyright material will stay in how exactly what exactly this is going to look like i'm not sure yet yeah how much uh have the the big American AI companies been involved in this process? Have you consulted with Meta and OpenAI and Google and how involved have they been in consulting on some of these issues? Well, they're quite involved. They talk to the commission on a regular basis in different formats. They talked to the members of the European Parliament and were involved.

24:58They talked to the national governments. They are quite involved. Yeah. Do you think that the negotiation of this act will have, until it's settled, will have a chilling effect? on on because right now we're in a period where uh everybody uh you know across the economy is trying to figure out ways to integrate uh foundational models into their products and services and in fact uh there are there's a sort of wave coming wave of new uh startups and applications built from the ground up on these foundational models. But if you're an investor or a company in Europe, it seems to me you would be cautious about investing too much in building products and services based on foundational models as they exist today without knowing what the regulation is going to be in a year or two?

26:20Well, I would see it the other way around. I think what we are basically doing is saying the providers of the foundational models themselves are responsible for fulfilling these obligations. because otherwise it would mean that the smaller companies implementing the foundation models in their products would be liable for fulfilling the whole legislation. And the legislation already says if you are in a high-risk environment, you have to make sure, for example, that the data is representative. How are you supposed to make sure that the data is representative if the provider of the foundation model just says, well, I have no idea where this data comes from.

27:12So I think what we're trying to do is to make sure that the burden of fulfilling the legislation is with the legal subject that can actually do that, has actually access to the data and not with somebody just buying it or implementing it that doesn't really have the possibility to influence what happens. And usually what companies want is legal certainty and not a completely open space. I mean, in the U.S. you have the same problem because if there's no legislation, companies are still liable in case of discrimination in lawsuits. So we have a different system in Europe. We say, okay, we give you the legislation, but then if you fulfill the legislation, you don't have this enormously high risk in lawsuits which have companies in the US.

28:07So it's a little bit different. But legal certainty is certainly something that companies in Europe do want and making the people liable who actually develop the tool and have access to the data and can have the possibility to know which data have been used, I think that makes sense. I don't think there's a chilling effect. I think there's more of a chilling effect if you know well that probably there's something coming but we don't know what it is it's better to know what it is yeah um and and the um are you consulting i mean i imagine you are with the u.s congress uh because everyone's following this very closely and and uh the u.s congress is looking at legislation itself to to uh to set a legal framework around some of these things how much uh influence do you think the the parliament's having on what's happening in the u.s congress i'm i have the feeling i haven't been consulting personally directly i've done that on digital services act um where i was i was really very very directly involved um and what i heard then was well go ahead in europe we are super interested in what you're doing and we try to follow your lead but we have difficulties in the u.s putting together majority on these issues yeah and it's actually more difficult to do if if the companies are in your country and they are extremely powerful corporations so in a certain sense we have that advantage in europe i mean we're sorry not to have the technology produced in our in enough on our continent obviously but it does give us some leeway uh to to reflect on the legislation and to pass on the pass of legislation what i'm hearing from my colleagues who have been to the u.s recently is that colleagues in the u.s are still looking very much at europe and what we are doing and that they appreciate the fact that it's it's a very dynamic kind of legislation and then let's see what happens yeah when you say dynamic uh you you've used that word a few times what do you mean that that it can change uh over time exactly yeah that for example on the foundation models what we are saying basically is not they are banned, they're not banned, or they have to do this and that.

30:51We are just saying providers take responsibility for what you're doing. So we asked, that's why I read out the paragraph on the risk assessments. They basically should themselves try to identify the risks and propose risk mitigations. So we're saying, well, you're rolling out this product, great, really interesting innovation looking forward to using it but please tell us what does it entail in terms of risk which is exactly what you would tell pharmaceutical companies so you're helping people get cancer therapy great stuff we're supporting you but tell us what the side effects are and if you don't do that you get oxycontin or something like that you know this is we are just asking these companies to do what in every other, each and every other industrial sector would be completely obvious.

31:48You know, give us the product, very interesting, but tell us what the risks are. I mean, we do drive and we know it's dangerous, but we do have traffic lights as well. So what we are doing here is just, you know, tell us what kind of traffic lines do we need? What do you think? And then let's discuss it. So I think this legislation, especially on the foundation models, will be evolving. But when you have an industry that's telling you, well, these are very interesting products, but we don't really know what they could do. I think we need to get involved as legislators. say well try to figure it out and let's discuss it together and figure out how how we can protect our societies together and and use this program these products in a way not to harm people yeah uh has the so-called threat debate that's cropped up in the last couple of months uh affected i mean i imagine it has um the uh the direction of the act and have you had because a A lot of the people who are raising these concerns are originally European.

33:02Yan La Kun from France and Yoshua Bengio from France and Jeff Hinton from the UK. I'm not sure what their current nationalities are. They've lived in Canada for quite a while. But have those people been involved in talking to the parliament? And the reason I ask is there's a discussion, you know, that debate has split the community, the research community. but there is an effort right now to come up with a consensus within the research community so that it's not just people sort of yelling at each other on Twitter that that there's some some unified view of what the risks are and how they can be addressed i don't think the fact that they are europeans make a difference i mean a lot of the people developing and also financing this are europeans so the fact that those who point out the threat are of european origin i don't think it made it made a difference absolutely right well go ahead i mean but have have you consulted with those senior researchers people like uh lacun and benjo and hinton i personally haven't i don't know whether whether colleagues did so i i haven't seen that as a formal process in the european parliament and i i haven't personally, but I cannot exclude that other colleagues have done that.

34:53I don't know. I imagine they might have spoken to the Commission as well, but I'm not sure about that. Yeah. What I do think is that more the discussion on the singularity issue and AI wiping out humanity has not been extremely powerful in Europe as far as this piece of legislation is concerned. because the first draft of this was written two years ago and is really trying to address the current very real risks like uh like bias for example um now i think something i care a lot about is disinformation also which is is not that well addressed in the ai act actually but i think bias spies were the big thing, security issues, that kind of thing.

35:48So more the things that are already on the table rather than what could happen in 10, 20, 50 years from now. That's more an intellectual kind of discussion also going on in Europe, but it hasn't had influence on this legislation, I think. yeah uh and and the uh where does the the how long is the process expected to take before this is formalized as law that is very difficult to foresee could be six months could be two years right i think there there is some ambition to conclude this within this mandate which ends in June 2024. So we have a year basically. But that would mean that the negotiations would need to end by something like the end of the year, January, February, in order to be able to go back to parliament and vote the final result and council the same.

36:49It's a complicated process, so we need a couple of months in the end of the negotiations to put it through. I think that's the ambition everyone has but it is a long piece of legislation with a lot of content and in an environment that is constantly changing at an amazing pace so i can't promise that this is going to happen i'm sure the commission and parliament would like this to happen yeah uh and and on the enforcement so the the the law gets passed uh who is charged with enforcement and i understand that it's up to the uh companies providing models to to ensure that they uh adhere to the act, but who is reviewing models or will be reviewing models and deciding on which risk category they fall under and whether they're in breach of the act?

37:53National authorities. So it's the governments of the member states who decide which authority will do what. might be one single authority it might be several i'm hearing in the netherlands it's going to be the data protection authority for example in in germany i think it's more what we call the agreed organism so they authorities that do audits on on technical specifications like in the automotive sector, that kind of thing. I think there's going to be a variety of models of implementation, which is going to be very interesting. This is why we also have the board, where these authorities will exchange on a European level on their experiences.

38:44Yeah. And what do you think would be the penalties? I mean, of course, it's too broad a question to ask and answer very specifically, but what kind of penalties do you envision? I mean, is it the sort of thing as with pollution, industrial pollution, that very large companies may be content to pay fines and continue to operate in the way that they do? or would they be banned entirely from the union? And in that case, how do you police that? How do you know if someone is using a foundational model that breaches the act? That's a pretty good question. I think we start out with fines. And I do have doubts about the efficacy of fines.

39:54We see that in data protection. That when Facebook, for example, was fined with a huge fight by the FTC, actually, stocks went up the day after. So, I do have my doubts about this. Yeah. it's also true that the data protection legislation has not been enforced very well in europe so we have a few fines but more are coming than the digital services act also works with fines so this is going on full speed next year and i think we will see some fines there maybe it's a mechanism that is not at full speed yet. And at a certain point, it will become a mechanism which does become not just cost of business, but a threat for a company, especially when you have shareholders watching this and saying, okay these fines are accumulating we are constantly um violating important legislation let's change so this is this is what i hope for um i hope i'm not too optimistic and uh what about uh government uses of these uh of this technology does does the act um constrain governments and militaries of member countries

41:35military um issues are not covered explicitly so military ai tools are not covered um it does constrain governments yes in many cases the rules for governments are even stricter than than for private operators because clearly governments have full control over people so people don't have a choice in some cases um the european parliament has um has made stricter rules also for private entities where the european commission only foresaw those strict rules for for public entities cases like for example data scraping in the internet with image recognition like clearview for example the commission only prohibited that for public actors and we added a ban for private actors too because i mean it's it's absolutely especially for women it's a super huge risk people you know see you in the street and can figure out where you live and what you done in the past 20 years of your life you know almost every woman has had a stalker at some point in your in your life and it's you know it's just a total nightmare the idea that there is a commercial software out there that allows everybody to know who you are just from from looking at you or taking a picture or something like that so but but normally the rules for for public entities for governments are even stricter than for private ones yes yeah um and i mean there's so many directions to take this there's a view that as this legislation takes hold and the technology continues to develop within those regulatory frameworks that different zones ai zones will emerge that you'll have uh the the the areas of chinese in chinese influence where uh you know ai uses in in surveillance will be much uh greater and and uh uh will have a much bigger impact on uh on governing society you'll have europe where it's uh it's less uh restrictive but yet more restrictive than the u.s and then you'll have the u.s which which will be kind of a a bit of a wild west uh depending on on what congress comes up with.

44:30Do you think that that's the case? Might be, yes. It's a very probable scenario. There could also be a scenario of the US and Europe merging. And the US may be following at least the spirit of European standards. that depends on the u.s i think yeah and as has been the case with gdpr yes yeah um and going back to the foundational models i mean this is a real problem for for regulators everywhere uh but in that that came along late in the process

45:26what's your your best guess of how that's going to evolve that existing models like gpt4 even though they cannot um

45:44you know with take out the data that that the models were trained on i mean it's too late the models are trained uh that those will be grandfathered in and and the these standards will be applied to models going forward or that those original models will eventually be banned because they they were not designed with all of these questions uh in mind i mean it just i'm uh and and i know not not uh no one can answer these definitively but i'm just curious with what you think well it would be great to have a glass for your words wouldn't it