Web-scraping bots

0:00Hi, and welcome to Tech Life on the BBC World Service, the programme about technology and how it's changing all our lives. I'm Chris Valance. This week we look at the impact data-slurping robots from AI firms are having on the online encyclopedia hundreds of millions of us visit. I am, of course, talking about Wikipedia. If you work in the fashion industry, if you are a fashion model, are you worried about AI? A lot are, and we'll be finding out why. And how do we keep children from seeing adult content? Many parts of the world are requiring sites to check the ages of their users. Now the world's biggest adult site argues we need a better system.

1:03Now when you have a question, where do you turn to for answers? Wikipedia, the free online encyclopedia, gets billions of visits every year. It's available in a wide range of global languages and amazingly it's all produced by unpaid volunteer editors. Lots of us turn to it for information and so too do the AI companies who send out programmes called bots to scrape information from the site, perhaps to help their systems to answer our questions. Recently, Wikipedia discovered a large amount of traffic it had thought were human readers was, in fact, AI company bots apparently coming from Brazil.

1:46And when they dug into the numbers, it also appeared its human readership had significantly declined, perhaps because more of us are turning to artificial intelligence to answer our questions. Selina Dekelman is the chief product and technology officer for the Wikimedia Foundation. I asked her what was going on. Well, what we've observed is over the last few months, since about May, we've seen a decline in human traffic. And we think it's about 8%, which is significant for us. And we believe it's primarily driven by automation. So bots that are seeking pages from our website and then using those to do things like train artificial intelligence or feed search engines.

2:32And is it that those answers, the answers that are created by generative AI systems are sort of getting in the way of people going to Wikipedia? So they're sort of, you know, they're taking your content, if you like, these sort of programs are scraping your content. And that content is being used by generative AI to create articles. And that means that people don't need to go back to Wikipedia? Well, because we can't observe what every person does on their individual devices, we don't know exactly. But what we have seen out in the world from reports from other publishers, industry peers, we know that other publishers like newspapers and websites, they're seeing declines in traffic as well.

3:22So we had anticipated that we would start to see a decline to Wikipedia. The other thing that we know is over the last several years, people have started using other kinds of media to get their news. So for example, short form media on a variety of sources. So like, TikTok, you know, YouTube, these types of things, people are using that short form video as well. So while we can say that it's mainly, you know, we might think that it's just AI, I don't think it's just that. We suspect that it's like a variety of things happening all at the same time. You didn't notice this at first. At first you thought your traffic hadn't fallen, but then you uncovered it, if you like.

4:06And that was all to do with some disguised Brazilian bots. You'd better tell me about that. Yeah. So we have these systems that monitor how much traffic we're getting, and they tell us things like where we think the traffic's coming from. And what we noticed in May was that there was a big spike in unusual but human-seeming traffic. So we investigated that. And as we dug deeper, we realized that this traffic was actually automated. When we say bots, that's what we mean is that there's some program that someone wrote and it's going in and like scraping data. So as we dug into that, what we realized is that our systems for detecting bots, they needed to change a little bit.

4:52So that's what we did. We changed our system for detecting automated traffic. and then we looked back at all of the data that we had since May and that is when we saw and detected this overall decline. And the way that we look at it is we compare year to year. So compared to the traffic that we were seeing in 2024, we're seeing about an 8 % decline. And we should be clear, these programs are basically sort of scraping the contents of your site. In the case of AI firms, that would be in order to get information to train their AI system or for their AI system to produce answers with. Yeah, that's right.

5:36You know, your listeners may know that there are a lot of companies right now that have gotten funding to go and try to create the coolest new AI tool, you know, that consumers will want to use. So there's a little bit of, I would say, a gold rush in this regard, right? Like people are trying to create these new tools. They're trying to get data as quickly as they can. And in that work, they're not necessarily thinking about what effect that might have on the sites that they're scraping. So something that we've been doing, you know, for Wikipedia and all the Wikimedia projects is we created a system for companies to actually work with us, call it Enterprise.

6:25And what we're trying to do is redirect some of that automated traffic over there so that we protect the services that humans use. People don't think about these things because the internet is just there. But, you know, there's a lot of computing that goes into keeping Wikipedia, one of the most popular sites on the internet, going and serving all the demands that are put upon it by human beings, but increasingly by these robots. And that costs you money. In fact, the type of requests they're making, you wrote, said that it costs you more money than other types of reading of your content, if you like.

7:03That's right. And I don't want to get too technical into exactly why, but we host our own data centers. We have a very small, very dedicated team of human beings that keep all those computers running. And just the way that the robots do their work, it's not like a human being. You know, the robot goes through and it looks at like every single page. It doesn't stop to, you know, chat with their neighbor or prove, you know, their friend at the pub wrong. You know, it just keeps like hammering through like every single page as quickly as it can. And that consumes a lot of computing resources. So it's just like a different style of access and it's better supported separately.

7:42So it's, you know, much better for the infrastructure if we can have these two, call them use cases, separated a little bit so that when the robot like wants something, we can provide it. And when the human wants something, we're very fast in getting them that information as quickly as possible. That's Selina Dekelman from the Wikimedia Foundation speaking to me from Portland, Oregon in the United States. And we'll have more on artificial intelligence shaking things up shortly.

8:21You're listening to Tech Life on the BBC World Service with me, Chris Vallance, and with you, our listeners. And we'd love to include your ideas and experiences in our programme and, yes, even your favourite Wikipedia article. We're going to be talking about AI in a moment and how it's affecting people's working lives. Is AI making an impact on the way you work? Or are you worried about it possibly replacing your job? Let us know and we can share your comments on air in a future show. You can email techlife at bbc.co.uk or you can WhatsApp us a text message or a voice note. The number is plus 44 330 1230 320.

9:05We read and listen to everything you send. And still to come, an adult industry viewpoint on keeping children safe from harmful online content.

9:24Now for some people, certainly not me, their face is in a real sense, their fortune. I'm talking about fashion models. It's always a competitive job, but many designers are turning to AI to create images of models, either invented or based on pictures of real people. In a moment, you'll hear from some researchers who've been looking at the issue. But first, we made contact with two people from the modelling world who've seen the effects of AI close up. The first is Shireen Wu, a freelance model who lives in Los Angeles. She told me her experience with AI. I had a very explosive story like two years back.

10:07It became viral on social media where I exposed a designer for replacing my face with AI. So initially I walked with the designer purely for exposure, so not getting paid. And so what happened was he then posted an image on his Instagram where it clearly looked like my body, but clearly not my face. it was actually replaced with an AI model of a white woman. I was really thrown off guard and spoke out about it and the rest was history. We should say that the designer mentioned there denied that he had altered the photo of Shireen. The impact of AI is being felt elsewhere too. Dawson Deng is a former model and co-founder of South Sudan Fashion Week.

10:56He said he's worried about AI leading to job losses. It's evident already and it's moving very fast. We're seeing a number of designers have already resorted to using AI to produce maybe their marketing content. So that excludes the other people they will have worked with to achieve all that. So I believe it kills a lot of jobs. My thanks to Dawson Deng in South Sudan. Well, as we mentioned, researchers have been looking into the impact AI is having on fashion models. I've been speaking to Zoe West, senior researcher at the Worker Institute, part of Cornell University's Industrial and Labour Relations School, and Alexandra Matescu, who is a researcher at the Data and Society Research Institute.

11:46With the help of workers' rights organisation, the Models Alliance, they've been talking to models about their thoughts on AI. The first voice you'll hear will be Zoe. So we've seen a few different ways that AI tools are being implemented in the industry. One direction that we see is kind of the amplification of editing tools with AI editing tools. And so the ability to manipulate an image of a fashion model just becomes much greater that you can radically manipulate those images with the advent of AI editing tools compared to the capacity that existed before. We're also seeing the use of 3D body scanning.

12:33And so the ability to create this, basically what could be used to create a digital avatar of a real flesh and blood fashion model. So that then if you have 3D body scanned this person, then you can use their image and create poses of them in all kinds of ways. And then the other direction we've seen is the creation of synthetic models where it is not based on a specific human model, but the training data is used to create a completely synthetic model that is then used as the brand wishes. And Alexandra, is this happening right now? If you're browsing pages of fashion on the internet or in terms of people selling clothes and things like that, is it being used by companies right now?

13:25Oh, absolutely. There was a bit of public backlash recently from an ad in Vogue magazine for the brand guest that featured an AI generated model that people noticed precisely because it had a sort of uncanny quality to it. Her limbs were a little bit off. There's awareness of it when it's noticed, but if it's more subtle, often it's not noticed. Zoe, what's the impact of this on the work of models and what are some of the negative consequences of this? So there are already a lot of practices that before AI arrived were already created difficult employment circumstances for models. So you have, for example, models tend to have very little control over their contracts.

14:15Sometimes they don't even have access to what's in those contracts. The management companies typically exert a lot of control over their work. And so you have a lot of murky contract situations that models aren't that aware of. And then when you have AI come into the picture, the use of AI is sometimes not even in the contract, or if it is, the terms are not really clear. You have situations as well where violations of image use and the terms around image use are very common in the industry. So it's pretty a widespread problem for models to find that an image taken of them for one ad campaign suddenly is used in another ad campaign maybe years beyond the terms of use have expired.

15:07And this is a way for brands to not have to pay for it again. And when you have AI editing tools come into the picture, it becomes far easier for brands to do this. And because you have an industry with such sharp power imbalances, a lot of models find that they don't really have a way to challenge it when brands are reusing their images. So people can end up competing with themselves, essentially. Exactly. Exactly. I was going to add one phenomenon that we saw was the rise of so-called AI modeling agencies, where some modeling agencies now have a dedicated face board, which is the board that shows the available models, including the standard face board, including human models, but then a face board specifically for AI models.

15:57For these avatars, models that have avatars usually are compensated, you know, about one tenth or less of their usual day rates. And these avatars are tied to the agency. So, for instance, we talked to one model who was really skeptical about her contract, who said she asked the modeling agency, does the avatar belong to the model? Does it belong to the agency or does it belong to the tech company used to create them? I mean, there's been a lot of emphasis in fashion of late on increasing the diversity of the people shown in adverts, increasing the range of body types that are shown in fashion adverts.

16:33How does the advent of AI affect that? What did you hear back from the models you spoke to for your research? You have these sort of unspoken racial quotas, for example, that a number of models spoke to us about where there's a sort of limited number of people who are, say, of Black models or Asian models that they'll have spots for in a certain campaign or show. and that if you start to have a situation where companies such as when Levi's announced that it was going to increase diversity among its models by using AI models, you have a situation where they're basically boasting that they're going to meet diversity targets using synthetic models instead of human models.

17:17So there's a lot of concern that this is going to create fewer opportunities for models of different backgrounds. That's Zoe West and Alexandra Matescu. Zoe mentioned an example of AI being used by Levi's. And we should add that the brand has previously said that it's not scaling back its plans for live photo shoots, the use of live models, or its commitment to working with diverse models. And if you think your job is about to be affected by AI, we'd like to hear from you. I'll mention our contact details again at the end of the programme.

18:01Well now for a little bit of a warning. If you're listening to the programme with young children in the room, you might want to turn the radio or smart speaker off and go make a cup of tea, because we're about to discuss some topics you might prefer not to have to explain until they're a bit older. In fact, our topic is all about how we stop children viewing adult content. It's something many parents worry about, we know. In France, in several US states, and in other places around the globe, laws are being passed requiring online pornography platforms to stop children viewing their content by checking the ages of their users.

18:41Three months ago, the UK's new Online Safety Act, or OSA, made it compulsory for online pornography platforms to have strong age verification checks in place. Users have to prove their age before they can access adult content on a site. And the law is being enforced by the UK's online safety regulator called Ofcom. The implementation is being watched by other regulators and lawmakers around the world. For an industry view of how it's going, I've been talking to Alex Kekesi, head of brand and community for the Canadian-owned company ILO, which operates several adult sites, including Pornhub, arguably the most visited adult site in the world.

19:26I asked Alex if she thought Ofcom was happy with how they're implementing the new law. Yes, we're compliant with the law. We comply with the law in all the jurisdictions that we operate in. That does look very different in the UK than elsewhere. It's no secret that, you know, in many of the states in the US and countries like France, we've outright blocked access to our platforms in response to age verification laws there. That's primarily because of major privacy concerns that we have with the way that these laws are drafted. But what I really want to make clear is that, you know, at least from ALO's perspective, we're very much in favor of age verification as a concept.

20:04We think that there's great potential for age verification when it's done right to keep children safe. What we've seen, unfortunately, is that with the site-based method of age verification, which is kind of the only format that we're seeing in laws as far as they've been implemented today, both in Europe and in the US, this is not effective. What have you seen in terms of what's happened since age verification was made compulsory? One of the big things to note is some of the sites that have seen an uptick in traffic. Unfortunately, if we give a really conservative estimate, this is based on a similar web statistic, there's somewhere in the ballpark of about 240 ,000 adult platforms that exist, or sites that I would say service adult content.

20:50So far we've only seen that there have been, I think it's 69 notifications that have been sent out to non-compliant platforms, and not all of those have to do with age assurances, have to do with other aspects of the OSA. What I have to say is that I really do feel for Opcom, right? They've been charged with an insurmountable task. When you're talking about 240 ,000 adult platforms, and we've only seen compliance from a small handful, I just don't understand how any kind of government body or any kind of regulator could be equipped to effectively enforce that law, which is one of the major problems with site-based age verification.

21:31I mean, they'd say that over 6 ,000 sites, I think, have introduced age verification as a result of what they've announced. But is your point that there are just hundreds of thousands more sites that haven't? Exactly that. It kind of sets up this game of whack-a-mole, right? So let's say what we've seen now since the implementation of the OSA is that there's a number of sites whose traffic has grown exponentially. And these are sites I should specify that are not complying. So these are sites that you can find very easily just by doing a quick search. That is very easy access. And again, this is essentially no barrier for anyone to access, be they minor or adult.

22:09And then one of the big concerns that we have there is what kind of content is on those sites, right? So from our perspective, our platforms are very safe. We have a number of different protocols and measures in place that ensure that the content that's on there is only from consenting adults. We have IDs and paperwork on file for everyone that appears on our platforms, in addition to a very robust moderation process that all of the content undergoes. That is not something that we can say for those websites, unfortunately. So we really are outliers in that sense. So you feel it's pushing people to sites that have fewer controls than you now have.

22:45And I'll say now have, because obviously you've been on a journey in terms of ramping up those controls. But you think that this is driving people to sites that are less regulated, less careful? Yeah, so this is things that we're seeing, you know, content being recommended, labelled as being potentially underage. Similarly, we're seeing content that is illegal because it's been ripped from OnlyFans. So a number of these websites are promoting content that has been effectively stolen from content creators where they're not able to control or monetise where it's landing. that's non-consensual content, right?

23:19So it poses problems on a number of different fronts. And you're exactly right. It's essentially creating this funnel where unfortunately, you know, users are bouncing away. We have the statistics there, which I can share with you today. We've unfortunately seen on Pornhub a 77 % drop. That is a massive number. So what we were really surprised and frankly disappointed to see here in the UK is that that number was so high because we thought, you know, of the things that we think is really wonderful about the OSA as far as it's written today is that it does offer less intrusive methods of age assurance than we've seen elsewhere.

23:55So one of the major concerns that we have is encouraging this practice, I suppose, of users handing over sensitive information so that they can access legal adult content. Like, I think that that's something that really everyone should take pause with. Like, that's something that's, you know, should raise a number of concerns, you know, on a very basic privacy and security level. That's Alex Kikasey, head of brand and community at ILO, owner of Pornhub. And ILO would prefer a system where age checks are carried out not by individual websites, but on a device. So, for example, when you buy a new phone, adult sites would be blocked by default and you'd have to prove you're grown up before you can access them.

24:37Well, that's ILO's view. We asked Ofcom for their response And Shona McCallum is here to tell me what they told us in response. Shona, what did they say? Well, Ofcom told us in a statement, until now, children could easily stumble across porn without searching for it. But now sites must implement robust age checks to protect children from porn and other harmful content. Sites that don't comply and put children at risk can expect to face enforcement action. They say they're investigating 62 services for potential failure to comply with the Act, and more investigations may follow. And they also say over three quarters of daily traffic to the top 100 most popular sites is going to sites which have age checks.

25:26Thanks very much for reading that, Shona. And that's TechLife's Shona McCallum.

25:36well we're almost at the end of this week's edition of tech life but there's just time to remind you that we'd love to hear your thoughts on anything you've heard in this program perhaps your job is under threat from ai or perhaps you've a view on how you can stop children accessing inappropriate adult content online if you want to send us a note our email address is techlife at bbc.co.uk or you can WhatsApp us or text us a voice message on plus 44 330 1230 320. Remember to include your name and where you live. Today's edition was produced by Tom Quinn and presented by me, Chris Vallance.