What are the dangers with AI making financial decisions?

0:12Simoney Kyriakou: Hello and welcome to the latest Margin Notes podcast with me, Simoney Kyriakou, editor of FT Advisor. As you will no doubt know by now, at the end of January, the FCA announced the launch of the Mills Review. This is a major consultation examining how advanced AI will reshape retail financial services by 2030 and beyond. In my opinion, it's probably even going to be reshaping it right now. But anyway, leading the review is Executive Director Sheldon Mills. He's highlighted many risks alongside the opportunities for AI, including things such as reduced consumer agency and unconscious manipulation and further decreased financial literacy.

0:51Simoney Kyriakou: as Joe and Jane Public increasingly delegate all their decisions to AI agents. But as our special guest today will explain, the long-term risk is not really that AI might get it wrong. You know, we've all heard about hallucinations. But the risk is that it could provide very persuasive decisions without that all-crucial empathy, without understanding the very humans who have to live with those consequences. So joining us to talk about this and other aspects is Greg Davis. He is head of behavioural finance at Oxford Risk and the author of Behavioural Investment Management. He set up one of the world's first banking behavioural finance teams at Barclays back in 2006.

1:33Simoney Kyriakou: And of course, Oxford Risk has been carrying out research that shows AI systems that are built without robust behavioural foundations can risk amplifying poor financial behaviour. This podcast will qualify for 40 minutes, CPD. You should be able to explain what the Mills Review aims to do, outline various areas where UK financial services need to embrace AI, and list some of the potential dangers for financial services providers if they get AI wrong. Greg, welcome to the show.

2:02Greg Davies: Thank you for having me.

2:03Simoney Kyriakou: Thank you. Well, as the Sound of Music once told us, let's start at the very beginning.

2:07Greg Davies: Very good place to start.

2:08Simoney Kyriakou: It is indeed. So can you explain the ABCs of the Mills Review?

2:13Greg Davies: Yeah, well, I mean, I think the Mills Review is the FCA asking a long-term question, which is how this new technology that we're all learning to interact with on a day-to-day basis might influence retail financial market structure and competition and investor behavior between now and the 2030s. So it's really looking at consumer outcomes. It's looking at market competition and market structure. and I guess it's looking at what regulation might need to be put in place in order to make sure that we can all move into this AI age a bit more safely.

2:48Simoney Kyriakou: Absolutely, but as AI is moving at a frenetic pace, we're coming up to agentic AI, it's all becoming very exciting and people are adopting things and adapting things very quickly. How quickly is regulation going to move to, is it going to play catch up all the time?

3:06Greg Davies: Well, I suppose in some sense, we're always playing catch up in all aspects to technological advances. Where I think we're quite lucky in the UK is that a lot of the existing regulation is already in quite a good place to deal with that because it is principles led, not rule led. The minute the technology changes or the world changes, you're in a position where if your regulation is very, very rule led, your rules are all not fit for purpose. Yeah, to change everything. If your principal is led, it is also with consumer duty, it is already outcome focused. And that I think is vitally important because one of the things that the use of AI is going to change is it means that the things that we have to validate are no longer individual outcomes from the advice.

3:53Greg Davies: I've recommended this or I've suggested this. It is how are those being used over time through processes and through systems. And how is that changing investor behavior, not just because they've received an outcome, but because they are continually and dynamically engaging with AI repeatedly over time? So the outcomes focus of consumer duty, I think, already puts us in a good place. But I also think that the regulation is going to need to think very carefully about the guardrails that are put around firms deploying AI. So can we make sure that there's safety guardrails so that certain of the problems or potential problems with using AI are curtailed at the outset?

4:37Simoney Kyriakou: Of course. We've got two sort of separate areas really, haven't we? We've got the sort of financial services firms regulated by the FCA that might be using AI or will be using AI. But then you've got chat GPT and other sort of generative AI that any consumer can get hold of and start asking questions. And we've seen new stories of how teenagers have done this to devastating effect because they've been getting mental health advice off of chat GPT. The worry, of course, is that bit is the unregulated aspect. So let's let's let's keep focusing on the regulated aspect that firms are doing then. What sort of questions should companies be asking as agentic AI becomes normal or just as it's sort of coming into play?

5:24Simoney Kyriakou: what should firms be asking now in light of the mills review?

5:29Greg Davies: There are a number, I think. The one is, what is the system designed and allowed to do? Because if we put in place an AI system and what it is doing is encouraging things that aren't necessarily in consumers' interests, like it's encouraging frequent transactions or frequent engagement, etc., rather than encouraging actions that will lead to good outcomes, Then you've got the AI might well be giving the right answers for its objectives, but the objectives are misaligned to consumer outcomes and to consumer safety, if you like. So what is it optimizing before? What is it allowed to do? Crucially, can I see the why?

6:09Greg Davies: So if I get an answer from it, am I able to determine whether that answer is valid, whether it is suitable, whether it is compliant? And this very rapidly gets us into the world where we can't just be turning on ChatGPT and getting a black box answer that is not auditable, that is not trackable, that is not built on an underlying rules-based model where we can see how the inputs affect the outputs. What happens in circumstances of market stress? This is when it becomes most relevant because people get emotional when markets get stressed and their portfolios are dropping and they're losing money.

6:48Greg Davies: That is where AI used badly could really exacerbate harm because it will precipitate and trigger more action rather than put frictions in place and help people to slow down. And the last one I would say is who's accountable for it? so the AI is giving the answers but who actually is responsible for the answers that are given and for for making sure that the use of AI has the guardrails and the safety mechanisms put around it

7:14Simoney Kyriakou: sure so if I'm hosting an AI sort of chat on on my site and I'm allowing my consumers maybe my maybe maybe I'm a financial advice firm and I've got a really good AI I've tested it I've looked through all sorts of scenarios I then sort of let it out to the public, they can interact with it on my website my advice clients are probably still going to come to me unadvised or maybe people I don't speak to or interact with very regularly are going to use the AI so they can use the AI and they can ask a question markets have fallen 3 % markets have fallen 3 % what should I do with my portfolio? Now, you may have tested an answer that says, well, the important thing is to remember your long-term goals, blah, blah, blah, blah, blah.

8:02Simoney Kyriakou: But then what happens when we start with personalization when people then say, oh, yes, but I'm really worried and I'm really going to need my money? What does the AI then do? Does the AI say, oh, I'm sorry, I understand that you are concerned about your money and you want to take some money out. Here is how you can start taking money out. How do you then prevent that kind of personalization from then giving the wrong answer?

8:23Greg Davies: Yeah, personalization is fundamentally important here. So different people will respond differently to those market events. And it is important that the system is set up to respond differently in those circumstances. So one of the things I think this exacerbates is a notion of behavioral vulnerability. The consumer duty speaks about vulnerability. It references behavioral vulnerability, but most of it is about vulnerability of cognitive decline, of financial vulnerability, of not having enough knowledge and experience. But in times of stressed markets, you could have someone who is perfectly financially well-off, who has no cognitive decline, who knows what they're doing, but they have a constellation of personality types.

9:08Greg Davies: They're prone to anxiety. They're impulsive. They have low confidence in their decision-making. That means these people are likely to respond in ways that are detrimental to them in times of market stress. And to me, it places the onus of the firm to have a deep understanding of their client's financial personality so that the system knows when it's responding to that question, it knows who I'm responding to. Just used I'm for an AR, but there you go. So who is the system responding to? And it should be differentially more cautious with individuals that it knows have certain personality traits that lead to impulsivity or lead to selling at the bottom or lead to freaking out when markets drop 3%.

9:56Simoney Kyriakou: But that assumes that that person is a signed-in unregistered user who has perhaps completed a psychometric profile or the advisor's completed one with them. if it's someone who's not signed in and not registered the system will have no way of knowing who that person is or what that person's personality is so i i think the default needs to be

10:18Greg Davies: almost engagement led safety if you like the less i know about someone and the less they are engaging with the system unless they are filling in the profiling form providing the information that we need to know that we can give them a flexible response, the tighter the controls should be. So the answers that we give to people, people should be able to unlock more flexibility by demonstrating that they have knowledge, they have capability, that they are engaging with the system and that they are providing the information to know that I'm giving you the right answer. If I don't know what the right answer is for you because I don't have enough information on you, then we need to be much more restricted in the sorts of answers that the system gives.

11:05Greg Davies: And in a time of market volatility, that default thing really has to be, if I don't know anything else about you, it should be introducing frictions. It should be slowing people down. It should be trying to focus them on the long term rather than the short term. And that should be a hard-coded principle in the system.

11:25Simoney Kyriakou: You will need to speak to a financial advisor. You will need to have a phone call with one of our people. Even half an hour, we can give you half an hour, no fee, just speak to us.

11:38Greg Davies: Indeed, or pieces of communication that might not require a human advisor, but that are specifically geared around framing the problem in a bigger long-term perspective, are providing and equipping that person with the knowledge and experience that they might need in that decision point. I mean, of course, it depends how big the decision is the person's about to do. Am I about to, you know, trade£100 in my regular ISA allocation or am I talking about selling 60 % of my lifetime investable funds at this point?

12:09Simoney Kyriakou: Yeah, of course. But it's that FCA's focus on outcomes, as you rightly said, with the consumer duty and with the principles-based regulation. um surely um i mean the fca knows this but how can the fca really deliver that kind of clear pathway for firms um for ai because you know you know sheldon mills when he was talking about you know when he was launching this review he said no regulator has a complete picture yeah you know these are we've got unknowns known unknowns unknown unknowns you know could you talk us through some of these sort of challenges that the regulator is naturally going to face?

12:47Yeah.

12:48Greg Davies: So let's say known unknowns, the things we know are potentially problems, but we don't know how big the problem is. So the one is misaligned objectives. If the system is optimizing for engagement or for transaction activity, et cetera, that is going to be incredibly harmful, particularly in volatility. So we need to, well, the FCA needs to make sure that any system that is being used in a regulated environment, it is absolutely transparent and clear what principles are baked into that system and what it is optimizing for on behalf of the individual. Another one I'd say in the known unknowns category is overtrust and persuasive design.

13:33Greg Davies: We know people will take advice from a machine, right? And this notion that we only listen to humans, I think, is wrong. We increasingly do listen to the machines. And the worry there is that the fluent language that the machine delivers can feel like competence. It can feel like this is an answer that comes from a great area of depth, but I have no way of validating it. So unless the FCA is mandating that systems are built, are not black box, are built on deterministic foundations, underneath it is a model. The model takes inputs and reliably a set of inputs will produce the same set of outputs.

14:15Greg Davies: That is not AI. That is just standard technology. The AI needs to fit over the surface and can help people to communicate flexibly with this thing. But I think a principle that needs to be built in right from the start is the AI should not produce the answer. It should be personalizing the answer and delivering the answer. But the answer itself must be something that is auditable.

14:39Simoney Kyriakou: Sure. Something that's on the advisor systems that they have gone through a sort of a list of, I mean, it can't be exhaustible because you just don't know what people are going to keep asking. And people often ask different iterations of the same question expecting a slightly different answer. And AI does give those slightly different answers each time.

14:59Greg Davies: Exactly. Almost by design. So yes, I don't think we can test this just by asking lots of questions because as you say, there's always going to be questions that we haven't answered. I think the important thing here is that underneath it is a rules-based system. So if I'm asking how much risk should I take, we already have frameworks that enable me to understand how much risk is right for you and how is that influenced by every aspect of your financial circumstances and your preferences and your personality, et cetera. That model should still be there. The answer that it gives you, that the AI gives you, the AI should never be making up the answer.

15:39Greg Davies: It's opening up a model. That model says, here's the answer. And the AI can then personalize how that's presented to you and how you are persuaded towards it. there's a risk even there though because even the right answer if it is delivered too quickly in too convincing a way and that right answer doesn't consider aspects of your personality you might push go on that answer and you might not be emotionally ready to live with it

16:11Simoney Kyriakou: but that's right that's what ox of risk has been looking at hasn't it that how that right answer might be given at the wrong time or lead someone to make a decision at the wrong time without understanding human empathy human emotions behavior yeah absolutely so is there something that we can do with ai to kind of give it a reason to try and show empathy or i mean it's really hard because it's a very human trait isn't it and we always talk about how the advisor is the the human in the process and it provides the empathy and provides that listening ear. But if people are going to bypass that and just use a system, you do need some kind of intuitive stopgap.

16:53Simoney Kyriakou: I mean, are there any sort of keywords you can use or what can firms do to kind of prevent that kind of emotional response from a human to something that's completely inemotional? That's not a word, but I've just made it up. Inemotional, we're going to use that one. Well, slow down for a start, I think.

17:11Greg Davies: It is important when, let's say, something has given me the right answer. It is the right answer for me to get to eventually. If I suddenly push go on 100 % of that answer right now, it could have all sorts of knock-on consequences. So if what I'm trying to look at is good client outcomes, the machine needs to be built not just to stop at analysis and go, here's the right answer. The machine also needs to have built into it consideration of financial personality and on steps along the journey. So the right answer itself becomes not just the financial thing to do. It's the financial thing to do that is also sufficiently emotionally comfortable for me to be able to act on and maybe to be able to stick with it.

17:59Greg Davies: That means that some of the stuff that we've been working on for years at Oxford Risk, understanding financial personality becomes a crucial input into the system itself. It's not just, oh, we'll arrive at the financial right answer, tick all the boxes. It's suitable for your financial needs in the long term. Go. the actual answer the system puts out needs to go through a layer that says what do i know about this person's financial personality there are likely emotional and behavioral blockers to putting this into practice and the things that could be harmful if they do it too fast particularly in times of stress or in times of volatile markets can it be giving them a cooling off period say

18:42Simoney Kyriakou: like within 24 hours we're not going to execute this trade we're going to give you 24 hours to think about it. I mean, that's one way of doing it. But then if the markets considerably move against you in those 24 hours, then again, you could then say, well, hey, I told me to wait 24 hours. And now, look, I'm even worse off. Thank you very much, mate.

18:59Greg Davies: Yeah. I mean, that, of course, is true of a human advisor as well. And I think as long as you can point back to the principles that you've put in the system that are there on balance to get to good client outcomes, then I think that's appropriate. And you do, you know, pause points are, take away all the technology in the AI. Pause points from a behavioral perspective are one of the most efficient decision tools that any of us can use. Just, you know, never push go the minute the answer arrives in your head. At least go and make yourself a cup of tea or something. And these frictions are ones that often in financial technology or technology in general, people have been trying to go, you know, UX designers, I want to have the minimum number of clicks.

19:44Greg Davies: I want to make it as easy as possible for people to get from A to B. In financial decision making, that can be incredibly harmful.

19:52Simoney Kyriakou: Yeah. And the banks have been doing that, haven't they, with a scam smart technology, you know, or you have to have another phone call, or you have to have another text, or you have to go through all these series of questions.

20:01Greg Davies: Yeah. Stop, get people's minds out of the decision, get their vision onto the far horizon rather than the here and now, enforce you to go and think about something differently or click a button. All of these frictions are very, very valuable. And they don't need to, in the end, stop people from doing things. They just need to make sure that people are given the opportunity to slow down and breathe before they make the decision. And the degree to which that happens should depend on that person's personality because it matters more for behaviorally vulnerable people. It should depend on their degree of engagement with the system because if they're not engaged, then they haven't built up the knowledge and the capability to do it naturally.

20:50Greg Davies: They're more at risk. and it should matter more in times when there is stress, either at an individual level because, I don't know, there's been a death in the family or I'm going through retirement or at the universal level because markets are in free fall.

21:06Simoney Kyriakou: I'm wondering whether it could be something that, let's assume it's a provider of products rather than a financial advisor. I'm wondering if a provider could have something up there that says, you are about to execute this transfer or execute this decision. Would you like a free phone call with a financial advisor? Can we signpost you to a financial advisor or can we signpost you to one of our experts? Would you like to speak to a human before you go ahead with this decision? It could also then just give someone, either the provider could signpost to a friendly financial advisor or signpost to one of their own trained experts internally.

21:45Simoney Kyriakou: that could help provide that human sense check.

21:49Greg Davies: An opportunity for pause. And again, of course, the provision of human advisors too much in that thing can get quite expensive. So I think it doesn't need to go all the way in every case to a human advisor. If you have triaged the people in front of you, you know these are behaviorally vulnerable, you know they're in a stress situation, absolutely. For people who are triggering fewer stress signals along the way. The pause might be something that doesn't have to involve a human, but is still sending them to go and read this, confirm that you've understood it before you come back and do it. So I think there's a whole raft of things.

22:26Greg Davies: The main point is we have to build them in consciously because the system is not going to do that by itself. It's going to be optimizing for whatever principles it thinks it should be optimizing for, which is mostly if I found what I think is a right answer, I'm going to persuade this person to do it.

22:43Simoney Kyriakou: Yeah. Yeah. Well, I mean, we're going through all sorts of testing ourselves, you know, asking questions of our own content and, you know, stress testing the answers and saying, well, you know, actually this needs to be done. This is not quite the right answer. But I mean, the danger is therefore that also financial advisors or providers, they'll have realms of information on their website. A lot of it might even be out of date. you don't want to get rid of your archive but that thing that ai that you're employing is going to scrape your own systems what sort of checks and balances should people be putting on their own systems to make sure it's only putting through relevant data relevant information i mean that's

23:29Greg Davies: that's a very good point i mean so the the the kind of ai layer that we're building at oxford risk is I don't know very much think about this as a layer on top of stuff we've already got. Because then you can start to control what goes into the stuff you've already got. And there are three buckets that it's sitting on top of. One is data. So data on the client that I know to be true and valid. And this will be true of most advisors. They will have data that hopefully, largely speaking, is up to date on their financial circumstances and their personality profile, et cetera. The other one is the actual models it's sitting on top of, so the input-output models.

24:10Greg Davies: And so the machine doesn't make up the answer. It goes and retrieves the answer. And the third one, exactly to your point, is the knowledge base that it's retrieving from. So you want to set up the system so that maybe the interface looks like ChatGPT, but it's not just going and asking ChatGPT the answer. It is going, first, look inside this box. That box is our knowledge base and everything we've put in that box, someone has had a look at. It is validated that it's up to date and they've put it in there. And it takes a bit of work to do that. But I guess we have this and most advisory firms will do the same.

24:49Greg Davies: We've been going through a process where we're looking at every single document and article on our website and sifting them into those that we think should go into the knowledge base and those that should sit outside. and it takes a bit of work but what it means is that the the system the ai is always reaching first for stuff that has been validated and read through and and and if necessary brought up to

25:13Simoney Kyriakou: date first and foremost yeah i mean there are two schools of thought here one is you know don't let the quest for the perfect get in the way of the good enough and you know sometimes we've just got to go live at this right now which is kind of my advice to my team sometimes um but then there's that other thing that you're pointing out saying well actually sometimes when it involves a financial transaction it has to be perfect yeah um i mean it's really interesting so it's again

25:39Greg Davies: strip aside all the technology this don't let the best be the enemy they're good is is one of the biggest things that gets in the way of investors we know that people all around the world and particularly in the uk sit on cash year after year after year because there are many very many people are so scared of not getting it perfectly right that they don't do anything at all. And actually, so that is absolutely right. It's not, you know, not doing anything because you're too feared of getting something wrong can be the most costly thing for you. And I think that's absolutely right here first. I would not suggest that we don't start using AI and we don't start experimenting with it.

26:18Greg Davies: we just need to pick which bits we do and we need to prioritize safety in certain areas first. Most financial transactions do not have to be perfectly right. What they have to be is not badly wrong. And so if we can set up the system to very stringently avoid the things that we know are deeply detrimental and allow more flexibility over the rest, I think that's probably the right approach to start going through this.

26:46Simoney Kyriakou: Sure. And you're working with True Potential. I was at the conference in Birmingham on the 27th of January. I found, I think it was a really, really interesting conference. And it's interesting to hear how you're working with UK financial advisory firms to kind of help them on that journey. Can you talk a little bit more about how you're helping and what you're doing, what Oxford Risk is doing? Yeah.

27:08Greg Davies: So a few things that we are doing here. One is, as I say, we are building this AI LLM layer over everything we've got. It's our existing technology, our existing system. So it's all being tested in order to invalidate it. But it's not always easy for advisors to interact with this and at an individual level, get a narrative of why this answer is right or why it fits with this client. So that is something that the AI and LLMs are very good at doing. It provides a narrative and an interpretation layer and talking points for advisors over the top of this complicated system that's all about suitability rules and risk profiling and risk capacity.

27:49Greg Davies: So it's to make it more accessible, more interpretable. And then you can start to say, well, whatever your answer that comes out of this system, your risk level, your asset allocation, the AI is helping you, the advisor, to explain to the client why it's the right answer. And it's doing so in a way that considers the financial personality of this person. So it will adjust the tone. It will adjust the level of detail, et cetera. So that's one thing is just better explanation and better narrative for advisors.

28:24Simoney Kyriakou: So the advisor can sort of deliver that personalization and then can look for those behavioral aspects, look for perhaps identifying vulnerabilities or identify anything that needs to then be fed back in.

28:37Greg Davies: Exactly. So the system is really feeding up to the advisor's talking points. Yes, here's the right answer and you've got your suitability letter. But let's explain to you in human language why it's the right answer. Let's identify where this person might have behavioral vulnerabilities. As a result of that, how should you talk to them? How should you communicate to them differently? So that's one thing we're doing. Another is imagine that tomorrow the markets drop 5%. Don't need to be imagining that. I mean, things are already going a little bit hate-shaky. It's not difficult to imagine right now.

29:12Greg Davies: Yeah. So every wealth manager around the world, when markets drop, they sit their chief investment officer in a dark room with a cold towel over their head and say, write me 600 words I can send to my clients, which basically says, don't do anything. Stay calm. Now, that's all very well, but those 600 words are typically written by someone in the investment industry, largely for other people who speak the same language. so we have developed a tool that can take seed copy like that and at a touch of a button it will instantly rewrite it for behavioral segments and behavioral personas that if people have been using our tools they know which of their clients sit in which of these personas and so it is taking communication that you know you want to do anyway but automating a level of personalization that means it's more likely to resonate and land with the people that you're going to be giving it to

30:10Simoney Kyriakou: Sort of like what we do as journalists. We take interesting, insightful analysis from experts such as yourself. And make it intelligible. Well, I wouldn't say intelligible. I mean, I'm the one who makes up words, as we already said, but we sort of personalize it for the advisor audience and say what this means for them. Yeah. So, yeah, I mean, I think that that's a fascinating and really useful tool. I hate the word tool, but, you know, what other words have we got for it? You know, something that can actually be used and be useful for advisors and for providers. Just going to kind of talk a little bit about that agentic AI.

30:47Simoney Kyriakou: We don't really know too much about this. And there is a concern that it could widen outcome gaps between consumer segments. Yeah. Unless those behavioral differences are explicitly accounted for. So what kind of behavioral foundations generally would be needed? and what sort of things should consumers generally understand before they interact with any kind of chat GPT, whether it's on an advisor site or whether it's just on their phones?

31:16Greg Davies: Yeah. So the agentic changes the, I guess, the threat level of the harm that can be delivered because the agentic AI means it's not just generating an answer. It can actually then formulate a plan and take actions potentially in a workflow for you. So it's almost a delegation tool. I'm not just asking it a question and getting an answer. It may actually be starting to act on my behalf. And that really matters because it's to some extent taking humans out of the loop. And it means that the harm, the potential consumer harm there is not just the answer being wrong, but that the answer is acted on rapidly without any safety checks and audits, et cetera.

32:01Greg Davies: So the first thing that we need to do in terms of doing that is to make sure we absolutely have an understanding of what permissions are granted to the system and what it can act on and what it can't act on. And as a consumer, if you don't know what this thing is able to act on on your behalf, then I would be very, very worried about that. But the way through that, I think, is to make sure that we absolutely know what is true in production, in production of that answer. So we need deterministic anchoring. We need to know that the system is actually producing an answer that is validated and verifiable.

32:43Greg Davies: we need to put clear limits on what the system can do and cannot do and provide audit trails of that we can expose to go back and say well you know was it allowed to don't give it access to your apple wallet yeah i've booked i've booked three hotels the same weekend in newcastle great yes exactly and then um yeah yeah no i mean it it is it the other thing is it turns the whole thing into something much more dynamic. Traditionally, advice, of course, is an ongoing dynamic system, but a lot of actual advice is delivered intermittently in a room at a point in time. And if the advisors are doing their job well, there will be annual reviews and you'll be checking into it.

33:28Greg Davies: But most of it happens at a moment that's quite controllable. When you've got these systems, it means that the frequency of interaction is likely to be much faster. I mean, I asked the AI yesterday what I should do and the markets have done something, so I ask it again. So the dynamic interaction frequency increases. And it means that it's much more the process of the journey that we need to be auditing and protecting rather than the answer or the actual solution that's provided at a point in time. And that, I think, is a fundamental difference because if what I'm going is, is this answer correct or not versus is this sequence of discussions and delegated actions over a prolonged period of time correct or not, that is a very, very different thing to be able to control.

34:20Greg Davies: And it's a very different level of harm that could result.

34:24Simoney Kyriakou: Does it come with an off switch?

34:26Greg Davies: Hopefully, yes. I mean, you would hope.

34:29Simoney Kyriakou: I mean, it's a simple question, but you know, These AI companies are so keen on rolling out all these upgrades almost without you asking AI overviews to just come in, AI discover. All of these things are coming in. And it would be nice if we can say, do you want to enable agentic AI? Yes, no, no thank you. And then you don't. You just stick with old AI.

34:50Greg Davies: This is, I think, goes on to that question of accountability and where is the accountability. And if a firm is using an AI system to be able to provide this sort of ongoing communication with its clients, it has accountability for client outcomes under consumer duty. And we'll see what new regulation comes out of, Mills Review, et cetera. But even with the current regulation, you would say if you and your firm have deployed an AI, you are still responsible for the outcomes that result from that. And therefore, you better have a pretty clear idea of what it is permissioned to do and not do at every single moment.

35:32Simoney Kyriakou: And do you think the FCA or the government is going to be able to police all these new companies that might just kind of come up and say, hey, I've got this new AI-based trading app or I've got this new AI-based compare the investments app. It just comes up and it could be based in Malta or Cyprus or somewhere. Just picking on those poor islands. They're great places to go for a holiday, by the way.

35:56Greg Davies: I think it's going to be difficult because you can put in place the regulatory principles and say, you know, foreseeable harm, consumer detriment, consumer outcomes, et cetera. And those can all be in place and people can be accountable for where that fails. But that's only exposed, right? You only know when the system has failed, once it has failed. And by that stage, a lot of the harm is done. And the area is moving so fast that I would share that worry that exciting tools come along. We don't dig deeply enough under the bonnet as to which keys we've handed over in the process of doing so. And that although the regulatory framework says, you know, this is not what you should be doing, do we across the industry everywhere have within all these firms the expertise to be able to unleash these things safely that is a genuine concern yes yeah well i think that's a really

36:56Simoney Kyriakou: good concerning note to end on usually i like to end on an upbeat note but i think that's an important one to to to end on so thank you so much greg it's been really insightful thank you just a reminder this podcast will qualify for 40 minutes cpd and you should now be able to summarise what the Mills Review aims to do, outline various areas where UK financial services need to embrace AI, and list some of the potential dangers for financial services providers if they get AI wrong. For now, thank you, Greg, for joining us.

37:27Greg Davies: Thank you.

37:28Simoney Kyriakou: And thank you for listening. Until next time, take care.

38:11Simoney Kyriakou: We'll see you next time. AI don't sound like everyone else. With Grammarly, you never will. Download Grammarly for free at Grammarly.com. Hey, it's Howie Mandel, and I am inviting you to witness history as me and my Howie Do It gaming team take on Gilly the King and Wallow 267's Million Dollars Gaming in an epic Global Gaming League video game showdown. Four rounds, multiple games, one winner, plus a halftime performance by multi-platinum artist Travi McCoy. Watch all the action and see who wins and advances to the championship match against Nioh right now at GlobalGamingLeague.com. That's GlobalGamingLeague.com.

38:49Simoney Kyriakou: Everybody games.