Could AI present CrowdScience?

0:00Alex Hern:This BBC podcast is supported by ads outside the UK.

0:30at Whole Foods Market. A burst pipe. A dead water heater. The AC calling it quits. Who do you call? HomeServe is an easy way to handle unexpected home repairs. With plans covering stuff, basic homeowner's insurance usually won't. Instead of scrambling for a contractor, you make one call to get the repair process started. Join the millions of customers who trust HomeServe right now. Go to homeserve.com slash podcast for 50 % less your first year. That's homeserve.com slash podcast. Savings compared to renewal price. Void in Florida. Hello and welcome to CrowdScience from the BBC World Service. The show that is powered by listener curiosity.

1:11What do you think of that one? I quite like that one. It's sort of got Morgan Freeman vibes. Yes. Okay. Have a listen to this one. Welcome to CrowdScience from the BBC World Service. The show that is powered by listener curiosity. I'm scared. Yeah, that sounds kind of spooky. That's like, yeah, CrowdScience gone evil. Should we have a British one? Yeah. Welcome to CrowdScience from the BBC World Service, the show that is powered by listener curiosity. He sounds quite disinterested, to be honest. He sounds quite bored. OK, look, let's stop playing around with these AI voices and we'll get you, the real human, to introduce the show.

1:51Hello, I'm Caroline Steele. And I'm Anand Jagatia. And you're listening to CrowdScience from the BBC World Service. The show that's powered by listener curiosity. This episode is about something a little bit close to home. Thanks to a question from a crowd science listener in Taiwan.

2:09Alex Hern:My name is Po-Chun Chen. I'm from Taiwan. My question for crowd science is, is it actually possible that AI takes over all human jobs? Is it possible that AI takes over, did you say, all human jobs? Yes, all human jobs. What made you think of this question? Are you worried about your job? I worry about the future. I'm just 27 years old and I wonder whether there are jobs for me to apply for the next 30 or 40 years. So you're only 27 and you're worried about the future of your career. Are there still going to be the options for you that there have been for other people in the past? Yes, yes. Are there any jobs that you particularly wouldn't want to be done by artificial intelligence?

3:00Alex Hern:I think radio show, for example, if the presenter is AI, I mean, that's a shame, of course, because I can feel the emotion of the presenter. Thanks, Po, for sending in your question. And I think a lot of people will be wondering the same thing. Could AI take my job and one day will it? And Poe specifically doesn't want a radio show like CrowdScience to be made by AI. The fact that we're even having a serious discussion about this is wild. Two years ago, you wouldn't have even thought of something like that, would you? It feels like AI has moved so fast in such a short space of time. Well, all I can say is that as long as we have anything to do with it, the CrowdScience team won't be being replaced by machines.

3:47but AI is here and rather than burying our heads in the sand or running away from it I think we can try to learn a bit more how it works and I guess understand what it's capable of and maybe what its limits are. Yeah which is a lot to cover in one show but we're going to do our best. Before we get stuck into all that let's start with some basics. So what do we mean when we use the term AI these days?

4:11Alex Hern:Overwhelmingly right now the technology that we're thinking of is something called a large language model, LLM. This is a conversation I had with Alex Hearn. So he's a journalist who writes about tech and AI for the British news magazine, The Economist. It's a technique that has been around for a while, where you take a huge collection of unlabeled text, scraped from the internet, books, podcast transcripts, and you, this is technical, throw it in a bucket and stir it with a stick until it learns to do maths equations. The thing you get when you finish training an AI system is a system that will take a batch of text and give you the next word in that text.

4:59So is a chatbot like ChatGPT a large language model or it's powered by a large language model?

5:06Alex Hern:A chatbot is a particular type of application built on top of a large language model. You might also hear the term foundation model or foundational model. That's very powerful in a lot of contexts, but it's also not a particularly easy to use system. So what you need to do is you need to take these raw large language models and shape them into something that is a little more like a consumer product. In ChatGPT's case, the big innovation there was a technique called reinforcement learning with human feedback. where basically you take this raw language model and you feed it a huge collection of questions and answers to teach it what a question and answer looks like.

5:49Alex Hern:And then you hand it over to humans who ask new questions, get new answers and tell it whether it did well or badly in answering them. And with that back and forth, you end up shaping this system into something that does what you want it to do. In this case, answers questions. OK, so that's how these AI models work. But Po wanted to know if they could one day replace all human job positions. And I guess for us to know that, we have to know if they are capable of doing a job as well as a human. Is that something that you can measure? Yes, and that is something we can do. Alex told me how much this has improved over time.

6:27Alex Hern:One of the metrics that I like to use to keep track of AI progress is tasks that an AI system can do with 50 % accuracy. So you set it on this task and it gets it right half the time. And they benchmark that against how long it takes a person, a software engineer, to do the same task. Pretty reliably, every seven months, the length of task that they can do has doubled. Now, the top flight systems can carry out tasks with 50 % accuracy that takes two hours for a software engineer to do. In seven months time, we expect that to be four hours. Seven months more from that, eight hours, an entire working day.

7:07Alex Hern:It will likely be able to do this task in 15, 30 minutes. But 50 % accuracy isn't great. If I completed my jobs at work with 50 % accuracy, I would definitely be fired. Absolutely. So for instance, if you want 50 % success, GPT-5 can do tasks that take about two and a quarter hours. If you want 80 % success, that plummets to 26 minutes. Okay, yeah. So it's sort of in a way it can do large amounts of things, not very reliably. Exactly. But the other thing that's worth saying here, right, is when I say something that takes someone 26 minutes, that someone is a trained software engineer. We're comparing them to professionals and we're saying a professional who is better than the vast majority of humanity at this category of task would take 26 minutes to do something that GPT-5 could do with 80 % accuracy.

8:06Those stats are actually pretty impressive. Those numbers are going up and up and up, right? Yeah, that's true. If it gets to 100 % accurate, then we're toast. So is there a limit then? Like how good can these models be in theory ever? Well, there's sort of two practical limits. So the first is the training data. So, so far, we've basically taught large language models based on all information and words that have been produced by humans ever. And that's got them really far, but we don't have another set of all the information and words that have been produced by humans ever. We're also limited by the computing power required for large language models.

8:49But if we were to overcome those practical limits, we also don't know if there's sort of a fundamental limit to the kinds of questions and tasks that AI could do because we're just not there yet. Wow. OK. Given what Alex knows about these models and the fact that he works for a magazine called The Economist, what is his take on whether AI will replace all human jobs? Well, let's have a listen.

9:13Alex Hern:It's tempting to talk in terms of jobs, but people who assess this sort of thing prefer to think of it in terms of tasks. It seems pretty likely that the task of mine that is proofreading the issue of an economist before it goes to press can be really, really helpfully sped up by an AI system. it seems very very unlikely that the task of mine that is taking a source out for a drink so that they like me more and tell me things that their employer doesn't want to say to me that that seems a task that is unlikely to be disrupted by AI anytime soon and so I think the question for most jobs is going to be which tasks that you have can be done by an AI or sped up by an AI?

10:01Alex Hern:And what happens to your job after that? How much power do you have perhaps to have more leisure time around your work? Or is the power with your employer to turn around and go, well, half of your job is now done by an AI system, so we expect you to deliver twice as much? Or to go, well, there's two of you doing this, so we now only need one of you. Or to go, we don't need either of you, actually. Those things are open questions, and they're more in the realms of economics and really sort of labour politics than they are a technological question about AI progress. I think what is really interesting about Alex's answer there is that the question of whether AI systems will replace a given job or all jobs isn't a question about technology.

10:44How well does the technology work? It's a question about humans. It's about what choices do we make as a society? Yeah. Do we want podcasts to have an AI presenter or do we value the lovely humans behind it. Exactly. Well, so that brings us on to Poe's second question, which is really about whether CrowdScience or a show like it could ever be presented by an AI. And as Alex said, it's easier to think about this in terms of tasks. And I can think of so many tasks that we, you and I do for most CrowdScience episodes, I think an AI would really struggle to. So meeting people and interviewing them and building a personal relationship.

11:23I got in a tank and played a guitar underwater. I'm pretty sure an AI would be short-circuited by that. Well, that's a good point. Like AI, it doesn't have a body, right? It can't go out into the world and like have physical experiences. Yes. Something else that we do on this show is that often actually the questions that we get sent in from people are questions for which there actually isn't an answer online. It's not that like an AI could just look it up for you and harvest all of the information created by humans so far. We're often asking scientists questions that they haven't really considered before.

11:57Like how much does the internet weigh? Exactly. That didn't exist online first. I guess there are tasks that an AI could do now. So it could do a probably a reasonable job of writing a script, especially if we trained it on all of the past examples of crowd, you know, 10 years worth of crowd science scripts. It might even do a pretty good job of being funny. I think we'd have to fact check it funny would be hard um humor that is something i think is really really difficult for ai because the kind of cultural context that's required is just so deep and i don't think ai gets it like if you ask a chatbot to tell a joke they're rubbish just like a knock knock joke yeah that is true and the other thing about humor is that so much of whether something's funny or not is in the delivery, right?

12:46Like how you actually say the thing out loud. Yes. And that's something we haven't even touched on yet. But if AI were going to replace me, it would need to have my voice, right? And that's what we're going to be looking into next. This is summer at its peak. Whole Foods Market Summer Fruit Fest is your invitation to eat the season. Fresh, organic, and bursting with flavor. Start your day with peaches and organic blueberries and yogurt. Build a grazing board with fresh fruit, prosciutto, and artisanal cheese. Then fire up the grill with no antibiotics ever proteins and fresh produce. Savor the season.

13:26Shop Summer Fruit Fest at Whole Foods Market. A burst pipe. A dead water heater. The AC calling it quits. Who do you call? HomeServe is an easy way to handle unexpected home repairs. With plans covering stuff, basic homeowner's insurance usually won't. Instead of scrambling for a contractor, you make one call to get the repair process started. Join the millions of customers who trust HomeServe right now. Go to homeserve.com slash podcast for 50 % less your first year. That's homeserve.com slash podcast. Savings compared to renewal price void in Florida.

14:09You're listening to CrowdScience from the BBC World Service, the show where real humans answer your science questions. I'm Caroline Steele. And I'm Anand Jagatia. And in this episode, we're answering a question from listener Po in Taiwan. So Po wants to know, could AI take my job? And I do mean my job. Could CrowdScience be presented by artificial intelligence in the future? And AI is something that the BBC are thinking a lot about. In fact, there's an entire department looking into how and if we should be using AI. So I reached out to them. I'm Nikki Birch and I'm the BBC's innovation lead in the generative AI programme.

14:50So we're trying to answer the question, could AI present crowd science? And I'm wondering if in your role you can make AI sound like the voice of a real person like me. Yes, that is possible now. I think the important thing from a BBC perspective is there is possible and then should we do this? And those two things are very distinct. I think it's really important that we keep the creative medium in a way that is about human interaction. OK, let's abandon the ethics for now and just say everything is allowed for this moment. What can be done? I can show you how it's done right now. It's kind of scary.

15:38OK, so I've been sent some audio of you speaking about three and a half minutes. And this is from a previous programme, which is used to then clone your voice. And there's all sorts of different sliders so I can make your voice faster or more or less similar to the model. but I haven't touched anything at the moment. I've just kind of put it in and then I've taken some text, which is actually a BBC News article about houseplants and how to keep them alive. I generated this. Serial houseplant killer. Here's how to keep them alive. Have you lost count of the times you've had high hopes for a pot plant, but despite careful positioning and diligent watering, it always seems to die?

16:16Well, you're not cursed and you don't need particularly green fingers for your foliage to thrive. you just need to know where you might be going wrong. Wow, okay. I think that's really impressive. What was your reaction when you heard that? I think I'm less impressed than you are. I mean, it does sound, I can hear myself in it, but for me it does still sound more like a robot than it sounds like me. It's recognisably you and it has features of your voice. It has quite a lot of warmth like you do, the pitch is right. It has the same amount of vocal fry that you do at the end of certain words. So I wanted to understand a bit more about how that kind of speech is generated and also why, as we've discussed, it sounds a little bit off.

16:57So I got in touch with a linguist, a guy called James Kirby, who is a phonetician from the University of Munich in Germany, and I played him the speech to see what he thought of it. So it sounds fantastic. I mean, if I didn't know that it was a synthetic voice, it would be very hard pressed, I think, to guess from a short excerpt like that, that that wasn't read by a professional presenter or professional newsreader. The main thing that really stands out to me is the second sentence where she says, are you trying to keep these plants alive, but they always seem to die? The pitch is sort of going down as though she's making a statement.

17:31But given the rest of the context, I'm expecting her to be posing a question there. And that is a function of kind of how these models are trained and put together. There's no understanding that's built into the system explicitly that says, this is now a question. And so because it's a question, I need to produce it in this way. So to oversimplify a little bit, it's sort of like you give them millions and millions and millions and millions and millions of examples of these things. And this has resulted in subjectively sort of more natural sounding synthetic speech. But these systems are also a lot more difficult to tune and control because there's no longer a kind of direct access to the features that they're using.

18:08It's an enormous black box. Whenever I hear about AI being a black box, that is when it scares me. I want to know exactly how it works so that I can fix it or so that someone skilled like a software engineer can fix it. Yeah, it's a good point. Like, obviously, this is a black box that just works, right? It gives really, really good results. but it also the fact that it can't even get a question right just shows that the black box model approach just doesn't sometimes do simple things that you want it to do. Yeah which is confusing. Yeah so I wondered whether as a phonetician James could break down the features of human speech that make it sound natural.

18:46We often refer to or use the term prosody which is an umbrella term that really refers to a combination of rhythm intonation and stress. so how we're phrasing the words how the spacing of the words is occurring where i'm putting emphasis and that's probably the most important contributor in some sense to our perception of naturalness because when humans speak we're constantly varying the duration the intensity i'm putting pauses between some words and then i'm using pitch i'm altering the pitch all the time in part to signal to draw your attention to particular words or parts of the sentence but also to signal what we call information structure.

19:24So am I asking a question or am I making a statement? Am I trying to provide emphasis? Am I trying to convey emotional state or some kind of nuance? Very often we're trying to do all those things at the same time. Interesting, because yeah, this isn't something that I ever think about day to day when I'm speaking, but it is something we think about a bit when we're presenting, right? And sometimes if we're recording a script and I'm presenting and you're producing, you might say, hang on, can you record that line again. Yeah, like as a presenter, part of your job is about conveying this extra level of information and nuance and emotion.

19:58So when we're recording scripts, often the producer will be like, no, actually, yeah, you need to put the emphasis on this word. And getting that right is sometimes tricky. So I wanted to see if there were any sentences that would trip it up. So I asked James and he gave me some example sentences to kind of test the model. So here's the first one, and it does a pretty good job of reading this sentence. The Prime Minister discussed transatlantic relations at a meeting with the US President last week. The last time he met with Trump was in 2018. So, pretty good. Yeah, sounds like me if I moved over into politics.

20:34And then if you just tweak the sentence very slightly, this is what it says. The Prime Minister discussed transatlantic relations at a meeting with US senators last week. The last time he met with Trump was in 2018. ah so if you were going to read that second sentence how would you yeah okay the prime minister discussed trans the primacy the ai is already better than me you're not doing a good job of keeping yourself in employment edit this out okay the prime minister discussed transatlantic relations at a meeting with u.s senators last week the last time he met with trump was in 2018 Exactly right.

21:13So this is what James predicted would happen. Basically, you correctly have stressed the Trump in that sentence because it kind of signals that we're no longer talking about US senators. We're now talking about something new. And so the emphasis there helps to draw attention to that. And James says that obviously this model that we've used currently doesn't take that into account. It's sort of treating those as two separate clauses that aren't related. But in the future, models probably will be able to take context into account. But there's another bigger kind of limitation that these models have, which we haven't talked about, that James mentioned to me.

21:49There is this issue of context dependency that we've been discussing. But I think it's also, while it's probably true for American English or Mandarin Chinese, it's maybe less true for Scots or Punjabi or Tamil or Welsh. In order for these models to be able to produce speech with this high degree of naturalness, it requires hundreds of thousands or even millions of hours of training data. And for many languages, most languages of the world, resources of that magnitude don't yet exist or don't exist at all. The accessibility of this technology is not the same for all people. OK, so it might in some ways be easier to have AI present CrowdScience in English than in, say, Welsh, because the training data isn't there.

22:34So maybe that's how we protect our jobs. Yeah, I mean, I don't speak Welsh, unfortunately. Chop, chop, get learning. So we've looked so far really at whether AI could present a show like CrowdScience. science and we've discussed that there are some things it might do reasonably well and other things that it can't do yet but maybe could do one day and other things that it basically just can never do but what about using AI presenters on other shows like are there parts of the BBC where this technology is going to be adopted? Yeah so this is something I spoke to Nikki Birch about we heard from her earlier and she's the BBC's innovation lead in the generative AI programme.

23:16There's certain things that we are not doing right now at the BBC. We don't have any AI presenters. There's no DJs that are just made up of chat GPT bots. We're also not generating any content that isn't human produced at core. So we might use AI to reformat text to speech, etc. But we're not using AI to create something entirely generated from the start. Are there any examples where sort of the BBC has decided this is an okay use of AI when it comes to presenting? I don't see this as presenting. Okay. But we are doing an experiment at the moment, which is using a synthetic voice to revoice BBC Sports articles about a particular football team, let's say Man United.

23:59So we've used AI to turn those football stories into a script. And then we've used a synthetic voice to voice that up. And the reason we're doing that is if you think about, ideally, all football clubs across the Premier League, you know, the lower leagues, every of them having daily bulletins all produced at the same time every day, 4pm at the end of the day. We actually couldn't do that with humans. It'd be too difficult. So it's looking at kind of personalised, scalable reformatting. The key thing is we're not generating new content. We're reformatting what is existing journalism, but just in a different model.

24:35I mean, this raises something interesting because what Nikki is saying is it's not a good use of resources or an efficient use of resources to get people to voice that stuff up. But it is something that you could get humans to do. Right. So, again, it's a choice. It's like a choice that's being made to use AI in a certain way. Yeah, but it's not like there aren't loads of humans out there that would absolutely love that job. I also asked Nikki about what she thinks the future of podcasts and AI might look like. We're talking about it in terms of production, but actually I think the next step will be about audiences and how they use AI to listen and to access crowd science, for example.

25:18So they might decide they only want a 10 minute version of the radio show and the tools will allow them to receive it in 10 minutes. They also might want it in Swahili and they can get it in Swahili. They might want their voice presenting it. All those things, the AI tools will allow, it's sort of what we call liquid content. They will have the tools to basically listen to crowd science in any way they want. And that, I think, changes things quite dramatically because at the moment, you and Anne are presenting and working on something very, very carefully, but that may be slightly taken out of your hands and people will access it in a way they want.