Who Protects Patients When AI Gets It Wrong? | with Lucy Orr-Ewing, BOAC #77

0:00I'm pro-AI that works. I think it's very easy to say, oh yeah, it's great when it works, but oftentimes everyone has a slightly different opinion about what it means to work or what it means to be safe. I think about something like social media, where social media was arguably one of the largest uncontrolled studies on young people ever undertaken that nobody signed up for. And we see an incredible amount of patient correspondence where people say, please don't make me talk to your robot. Please don't make me talk to your AI. I'm in pain. Please just give me a person. I think if you're a regular Claude user or ChachiBT user and you would put back in the 2022 version of the first release of ChachiBT, you would think this is so slow and awful, I'm never going to use this again, this is bad sushi.

0:42But actually at the time it was transformational sushi. And yes, we might have a terrible time with AI at the moment. Maybe like there's a hallucination or there's like a dysfunction of some sort if we're using a chatbot or maybe they miss a diagnosis or even worse, like that is the equivalent of food poisoning. And I hope that error is not so bad that actually it removes that person altogether from ever experiencing or wanting to experience AI again.

1:29voices, perspectives from outside the healthcare provider, provider operations space. And so today is one of those episodes. We're here with Lucy Orr Ewing, who has a very interesting background that I will let her talk through because I don't want to butcher it in any way. Lucy, thank you so much for joining us. Would you mind giving us the two-minute version of where you are right now, what you're focused on, what you're doing, and then we'll sort of go from there? Sure, I can talk about right now. And then maybe we can tie it to some of the context from where I've come from before. But yes, I'm Lucy, I am a director of policy and strategy for the Coalition for Health AI.

2:07Chai was, although I'm not a healthcare operator, we were started, we're a coalition that was started by eight health systems to begin with, Stanford, Mayo Clinic, Duke, John Hopkins, and others. And then we were able to grow to about 200 or 300 health systems now form this coalition and so you know we're a small team but we have a kind of mighty force of kind of healthcare delivery organizations behind us and we build responsible AI resources tooling frameworks playbooks I need kind of the policy side of that I have a policy work group of about 150 professionals who sit across kind of you know vendors healthcare systems etc and we try and build kind of consistency around how to think through the kind of policy implications for doing responsible AI.

2:55So I'm sure we'll dive into kind of what that means. But right now, you know, really thinking about humans have been nefarious for a long time longer than AI has been around. And yes, there is a lot of well placed fear around AI technologies and non deterministic outputs and all that. And I'm not saying that those are not misplaced. But I do think that We have analogous infrastructure to look after humans that are also fairly unpredictable and dangerous at times. And so, you know, me and the policy work group and the rest of CHI were thinking through how to build infrastructure to be able to trust the technology so that we can actually kind of capitalize on the benefits and reduce the harms.

3:39So that's where I'm at now. I have a lot of other kind of hats that I wear, including kind of being on the board of Utah Sandbox and being on the board of kind of FSMP's Task Force into Regulating AI. So just coming at it from different kind of angles. I work in the US. I go back and forth between the US and the UK. So, yeah, I can bring a kind of international view to some of these problems as much as that's useful. But, yeah, really excited to kind of dive into it. Thank you so much. And Lucy, you sort of pushed on it before saying, talk about what we're doing now and then maybe connected to some of the previous stuff.

4:15That is my next question. So it's an interesting time to be doing what you're doing. Well, it's the only time really you can be doing what you're doing. This wasn't an opportunity five years ago in the same way, I guess. But what was the journey to here? And how did the piece, I mean, in hindsight, everything makes sense. But how did the pieces sort of come together to lead you to thinking about the infrastructure to enable adoption, safe adoption of AI, specifically in healthcare, but arguably more broadly than that even? What was the journey to this point in time? um yeah you're right in that like you know journeys have to make sense retrospectively but at the time I sort of uh have found found it fairly kind of randomly but I think like if we go way back I was born into um you know a family of like public sector entrepreneurs um and MPs and kind of healthcare uh you know doctors um so I'm the youngest of four by about like 10 or 15 years So, A, you know, learn to kind of try and be the loudest in the room from an early age to kind of get noticed.

5:21But B, I was the last person of my generation. So the generation below me is kind of Gen Z, a sort of generation characterized by kind of reliance or over-dependence technology, some might say. and so I feel like I've got a very um I'm sort of the last person that has a childhood that was sort of on of an unfettered boredom um and a really kind of a good appreciation of when it is like safe and appropriate for using technology and when actually just letting your brain you know run wild um as we were used to when we're younger so I feel like and I don't I really see that in the years below me uh as not being the case and so I think I've got this like nice appreciation for like the benefits of technology and what it can do for sectors especially under loads of pressure like healthcare um i used to be chief of staff of the national health service in the uk and just seeing the pressures that we're under especially in the uk with the economy that we have um ai kind of emerges as like one of the only uh ways to kind of keep up with that demand but also that really um fine appreciation for like over-reliance is inappropriate but under-reliance is also inappropriate so it's that really like you know AI safety then kind of you arrive at AI safety quite quickly in terms of how to use it when is it appropriate to use it and I feel that has been informed kind of where I sit in my family the type of family that we are we are all kind of working in like the public sector or advocacy lobbying um in kind of different across different sectors um and so yeah my sector is healthcare but um I just think the the problems that I work on a daily basis they're people problems they're problems around like how to get humans to thrive in the age of AI and technology how to use AI safely uh on patients or patients how do they navigate you know transparency etc like there's just they're they're people problems more than anything um I did classics at uni so I'm not a I'm not a medical medical person I'm not really a technical person either um I'm just a kind of practical person I think those that are the best policymakers because they're the ones that listen to the experts and they write things that make sense so everything we do at CHI but especially within my work is just how can we bring like sense making into decision making in within healthcare and AI how do you build analogous kind of trust infrastructure that we have elsewhere in our world and as a consumer um how do we bring that into the kind of healthcare world so yeah in my kind of that's why it's a life story how I've arrived at the work that I do and like my lens on all this like how do you um you know thrive in the age of technology without over relying on it um I think it's my kind of view I'm actually quite anti-AI.

8:08I'm pro-AI that works, let's put it that way. Yeah, maybe that's the segue. First of all, interesting to hear how it was written, it was a result of genetics and environment that you ended up in this, I guess, in the role that you're doing right now. And knowing how we were introduced, I can see how the story continues elsewhere, perhaps in the family as well. I think your statement at the end there is an interesting one so you're you're pro ai that works or you're pro ai you know when it's safe or you're you know you're what is that actually so can we unpick that a little bit like when you think about pro it working or pro it being safe how do you even go about defining what success looks like here because it's i think it's very easy to say, oh yeah, it's great when it works, but oftentimes everyone has a slightly different opinion about what it means to work or what it means to be safe.

9:06How do you begin? Is that the starting point to this conversation about AI safety, AI adoption? Is it like, do we have a common view of what good looks like? Or what is the framework for even, what's the perimeter around this conversation? I think it's like, it's constantly evolving. So I think less about like, a line in the sand and more like a structural mechanism to be constantly comfortable with the the novelty of whatever's coming next so um as an example like you know when we buy a car we have some confidence that it has been through you know a level of assurance so that like when you crash if you crash hopefully you don't crash um the seatbelts are going to work you know you have and that's usually not done by a car manufacturer or really a regulator it's probably done by a kind of third party safety institution so like an underwriter labs or something similar um same reason as like why we're like comfortable putting like a light bulb in a room with like a newborn like we know that the light bulb is like safe um we might not know how that test is done or what the perimeter is for it to pass but we know that like it's there has been a process by which to make sure that thing is safe um I would say that like or you know there's a reason why I can eat sushi from a food van when I'm walking around a festival I trust that that food has been through a level of regulation and safety uh so that that has that that person has a license to give me that food I'm not asking for that certificate of regulation you know like as I just that's what I mean about being a practical person I think about like the way I live my life and what I rely on in order to trust something it's a it's a it's a suite of you know like the right regulation risk proportionate regulation for things that are going to be impactful things like food safety on raw fish um there's like reviews platforms for how can i trust that like my peers or my colleagues use something and they love it how do i kind of consume a technology and have confidence in its effectiveness through my community and that's especially like relevant within healthcare right like communities there are certain communities that have ridiculously low trust in AI so how do we work with those communities to like share the things that work within within those communities so I think and then it's kind of governance so if something goes wrong like if I receive poor healthcare or kind of any poor service I know that there's like a complaint system there's law and order there's like so many other kind of governance structures that we reside within as a as a human um and you know we should expect the same from health care we should expect like the same way that state medical boards and um exist and kind of yellow card and red flag red red card warnings for you know doctors that don't do their job properly there should be analogous infrastructure for ai that isn't serving its purpose so i think we should spend our time thinking about pre-market valuation like the minimum bar that it should hit for it to be for us to understand that it's safe and effective but I also think that we as like patients in the public need to expect this level of like multi tiered governance structures um that you know some of the ones I just mentioned uh so that this sits within like a very well established structure of trust so that's the kind of thing that I think about rather than just like it has to hit a 99 accuracy score because things like Like, you know, things are going to change.

12:44Inputs, output's going to change. I'd rather know that there's just like a net of services that are going to be able to capture these technologies when they no longer perform. It's interesting. My head sort of goes to two places. The first one is one of the early waves of healthcare AI stuff was around like AVT, ambient voice technology, and the sort of scribing. And there's a company in Europe who will remain nameless, but who I think are doing really interesting things that are a class two medical device approved Ambien Scribe. And I used to be involved in medical device development for laser-based technology for skincare, eye care.

13:20So I went through that journey, CE marking, FDA marking, et cetera. And I remember not only was there this sort of outward go-to-market validation process for the regulatory bodies, oftentimes third parties, but there was also the in-market regular surveillance and feedback mechanism to when there was some kind of instance in the field and how does that get pulled back and how does that get continuously assessed. And so it's interesting to me because getting class two certified or getting medical device safe and effective certification is like part of the journey, but not necessarily the full journey.

13:56The full journey is great. Now it's been like initially approved. Now, how are we monitoring, tracking, feeding that back in, et cetera. So my head goes, in one sense, goes that direction. But the other place my head goes, which makes me more interesting, is I think about something like social media, where social media was, you know, arguably one of the largest uncontrolled studies on young people ever undertaken that nobody signed up for. And, you know, it was all exciting in 2005, or whenever it happened. And now, you know, now we're living with the joys of, you know, youth depression and anxiety falling off charts in every direction.

14:27It's like, oh, yeah, we should probably regulate this thing. Whoops, for the last 20 years, where we didn't. So I'm just sort of curious, like, are we, are we going down the track right now? I mean, It gives me confidence to know that people like you exist and that your organization exists. But are we going down the track right now where we're basically doing social media 2.0, where we're saying the innovation is running so far ahead of the infrastructure, the institutions, the stability, the safety, that at some point there's going to be the boomerang that comes back and hits us all in the face.

14:56But at that point, A, people will have made a lot of money and B, it will be too late to put the cat back in the bag. Like, is that a real problem now or are we handling this differently or are you handling it differently? It's a very loaded question, but I'm just sort of curious from from the work that you do. I think the difference is in the social media analogy is that, again, I don't want to kind of call out names without knowing that industry inside out as much. so no healthcare, but they weren't really calling for regulation or even doing self-regulation themselves. They were pretty much like, you know, direct to consumer.

15:33Their algorithms were around, you know, how can I get people to stay on my platform for as long as possible? I think the technology vendors that we know, the ones that are members of CHI, so maybe that's a bias in itself, you know, they're funding a responsible AI non-profit, so therefore they might care about this a little bit more than the general. general um but you know they they have a certain amount of budget that they spend on doing self regulation or kind of self-evaluation safety evaluation and that's because the consumer demands it like if a health care provider you know health system cfo or even just like a doctor um or nurse a clinical professional i should say um they expect a level of kind of clinical effectiveness and safety study done on a technology that they use because they're going to be liable for something you know the ocr section 1557 rule um that says that everyone is liable for anything that you deploy in your technology that includes ai sorry anything that you deploy in your organization that includes ai and so um because of that rule there's a lot of stagnation with healthcare deployment of ai just because they need to like some level of assurance this thing actually works and it's not going to like you know blow up their machines or you know harm patients in in whatever way and so i think that like the technology vendors that we know they they're willing to like take on some liability like i think we've heard that time and time again like you know they want to be have a vested interest in um the safety effectiveness of their products and so they're willing to take on a bit of liability for that and not just you know leave it all with the healthcare deploying institution so there's that part of it there's also like I mean who knows if it's kind of safety washing green washing version of it but there is like a a recognition that we got it wrong in social media and that the equivalent now is to wait for a horrible harm to happen to patients and then retrospectively try and govern And I think everyone is trying to do this kind of proactive governance so that we don't harm loads of patients down the line, because like human lives are at stake in a lot more clarity than maybe social media at the time.

17:52I don't think anyone at the time thought that they were going to be contributing to mental health, especially in young people. Whereas I think, you know, harming patients through diagnostics, et cetera, like that's a very clear, you know, poor outcome. And so I didn't think, yeah, everyone's trying to make sure that never happens. So I guess it's a long answer, but I suppose like, I think preventative measures are being kind of prioritised because us as consumers and especially healthcare professionals are demanding a level of transparency from the technologies they're deploying. So Chai's doing a load of work across that space to try and open up that black box a little bit because otherwise technology vendors will not get adopted.

18:42you're talking about this idea of adoption. This is a market that I think everywhere in the world, I don't know if that's actually true, but I would imagine everywhere in the world is late to the party as it comes to adopting things full stop. It's often why, and even in some of the provider settings that we work and support, there are literally things still in the cabinet upstairs in the filing cabinet on paper. I mean, that is very much the state of play. And so there is this huge disconnect between what provider operations look like in certain parts of the provider ecosystem. Do you think adoption here is going to look different?

19:21Like, is the wave going to be different? Is the speed going to be different? Is it already different? Do you think there's going to be a, you know, we always talk about like the adoption cycle, hyper early adopter, and then early majority, is that chasm going to be much wider here? Are we going to have an early hungry audience and then this sort of pullback, a bit of AI wintering before it starts to take more adoption. How do you hear this in the conversations that you're in? How are people thinking about, I'm going to be frontline or I'm going to actually let other people make some mistakes here and then jump in on the second release?

19:53How are you seeing that play out from an adoption mindset perspective in the provider organizations that you work with? yeah i mean i'm fascinated by the states for two reasons like one is um you know it's a fragmented market like these are private health care companies that are competing with each other so they are more incentivized to use the latest and greatest tools because if they can get greater productivity gains than their peers then you know they will win the day um whereas you know nhs like the context that I'm from um they are more incentivized to like keep the status quo than maybe um in the US because they don't have the money to do prevention like yes we are one of the biggest value-based care systems in the world but we don't have the money to do prevention so we're just like this is finite resource that the NHS is and the more people that you know draw down upon it like later in the kind of care life cycle the more expensive it is so you'd expect us to be trying to push things to like primary care or even before primary care before a patient even comes to the gp but there's that takes time and that takes kind of money too i'm interested in the kind of the way that ai adoption will happen in the states because of the like experimentation of the private market but i'm also very interested in the legislative cycles of the states because you know coming from a policy background you know I can count on one hand the kind of bills that I've seen come through the UK whereas there's you know the state the way that the states are built you know they can pass bills and they are possible they're very overactive in the legislative space and especially you know in AI bills and then even a health AI bills there's several hundred a year and so I'm interested in that market when we see people like Utah who are doing you know i hope it's fair to say this because utah are great allies and colleagues of ours and i really you know adore the work they're doing um but they probably weren't known for being the most progressive state in that policy um or ai policy or even health ai policy like but they have become this kind of global leader in how they're thinking through um you know the implications of regulation on autonomous AI.

22:14And so I think both of those things are like, you're asking like, how do you think adoption is going to happen? I think it's going to be a mixture of like, you know, a non-linear kind of jump from like a regulation that opens up autonomous AI because, you know, people like Utah are experimenting with those kind of regulatory structures. You know, we all have a demand shortage. We all have access care, you know, shortages. um as well as kind of cost to varying degrees so anything that can kind of get there like get patients into care at a lower cost I think will will kind of you know hopefully bid up quite quickly but there's like a lot of things that need to be in place for that to happen so I think there's that I think like you know coming from NHS where we were talking about Ambient for like the last 10 years um only in the last year or so was when we saw that adoption i think that was probably from the u.s market we saw that actually the some of the nervousness that we had around people being recorded gradually the public became okay with that type of thing especially with the you know advent of chat gbt and other foundation models people came a lot more comfortable with the concept of being like constantly recorded um you know we're all using copilot and our teams meetings and our zoom meetings so that level of adoption came i think at the level of maturity that the public came into ai with you know i think they're not divorced from each other so i think there's a lot of things i think as we've become more used to the technology and how it works um not like coupled with like some some potentially game-changing regulatory structures that will probably come from the states and others will follow suit as well as like the u.s experimenting at a faster rate than um you know other markets so we can kind of see what works in other places and then as we become more comfortable starting to adopt ourselves i think it'll be a mixture of those types of things um not saying that it'll all come from the u.s i think there's fantastic things happening in the european markets i think um i just think due to the nature of how those health systems are structured.

24:29That might not be like the level of experimentation or the level of kind of commercialization of new novel kind of transformative ways of doing care in the same way. But not to say that that won't happen though. Now a quick break to tell you a little bit about Coherent Healthcare, the company behind the business of a clinic. Coherent helps fast-growing healthcare providers engage, support, retain, and nurture their patients from intake to discharge. As you get bigger, your patient body grows enormously, and it becomes much more than just another hire, just another front office team member. It's no longer a people problem.

25:07You need technology that can scale with you. Coherent helps you do that by keeping every patient on track. In short, we're the agentic CRM and patient support platform for outpatient hands-on healthcare. To learn more, visit us at coherenthq.com. Now back to the episode. I think I found myself thinking back to what you said earlier about sushi, which is you've said many things since sushi. But but you were you were talking about this this idea and you sort of came up again, consumer acceptance, maybe because of the chat GPT moment or whatever it might be. But, you know, you don't ask for people's food certificates, but you've now experienced enough of the food system to know that the person selling it to you hopefully has met the standards.

25:50And in a similar way, you now use chat GPT or you use an equivalent or more people use that. And so it's not as scary. You sort of have firsthand experience with someone recording you in a room. What we find, and maybe this is the other side of the conversation, so we don't do anything on the clinical side of health care provision. We're only on the sort of operational administrative side. And we see an incredible amount of patient correspondence where people say, please don't make me talk to your robot. Like, please don't make me talk to your AI. I am in pain or I don't want to talk to an agent or I don't want to talk to your AI receptionist.

26:26I don't want to, like, please just give me a person. And obviously, there are other people who don't care. So there's, of course, going to be bifurcation in that audience. When you are, you mentioned earlier, being in the room with 100, 120 provider or healthcare innovator partners that you're engaging with regularly, what do you think the mindset is from the provider base, is it, we need to provide both pathways. We need to provide the human first pathway, the AI first pathway. We like actually let's make it one pathway, but interweave these two things. Is it, is it an and, is it an or, how much of it is and, how much of it is or, does it change depending on the setting?

27:06I'm just sort of curious and I'm appreciating the US is probably a bit further ahead in that adoption curve. But equally, how far are we on the and or spectrum, I guess, in your view? What are you hearing that conversation? And similarly, to use the sushi example, you also sometimes get bad sushi. And so how long does it then take for you to eat sushi again? Probably takes you a couple months. You don't want to relive that experience. So are we starting to see the bad sushi moments and and do we need to recover is does that going to push people in a certain direction like i'm i'm i'm just kidding it's okay you guys are the u.s further ahead in that so where where are you how are you seeing that play out i think it's a in the first question like you've always got to design so if you're designing a hospital like an ambulatory clinic you know you've also got to be mindful that some people can't walk you know what i mean so You've got to be able to find a way for patients that don't have the capacity to access your services still can access them, even if it's like they can't get to the physical settings, whether that's like the ramps coming into the hospital or it's the, you know, phone service that sits alongside the care that you receive.

28:19Like, whatever it is, you have to design services that kind of are applicable to people that might not be able to access them in the way that the general populace will. So I think like translating that to AI and technology is like, there are going to be people, I'm not going to generalize, but tends to be like a younger, probably like more urban environment population who are really okay with an AI person calling them about their care or an AI, you know, first service. like maybe I get my prescription renewal done by an autonomous agent in the future like there are going to be people that are okay with that and then people that aren't okay with that and there's this phrase that was always used like meet people where they are but I think like classifying where people's um kind of comfortability comes with technology is really important and then designing services around that so like the National Commission in the UK have just done an amazing consultative process with the paternal public in the UK.

29:18They did 8 ,000 patients and they did 2 ,000 care professionals, which, you know, 10 ,000 is probably one of the largest kind of, you know, trust and AI type of surveys that has been done for at least the last couple of years that I know of. And they've been able to kind of wrap all of those recommendations into a really clear set of policy recommendations for the MHRA, which is our version of the FDA in the UK. And so I think like starting with consultation understanding what people's like problems barriers challenges are with the service and then designing the services around that so i'll give you an example like in mental health um there was a company that we i used to know quite well called wiser it's actually developed on like seven million people in india originally then it came over to the uk i remember like a lot of people had an issue essentially they were saying um you know a therapist is trained on a manual um and actually if you die if you kind of diverge from the manual um it's not actually a better quality of care you actually want to be like as close as possible to the training that you receive as a therapist and so that is a great use case for ai you can train ai to be a kind of chatbot version of a therapist um through the kind of training of that manual is i'm massively generalizing this career but just for the purpose of this example um and i think you know if that means that for 80 percent of people who wanted a therapist can't afford it it's too late in the night like to access a care or you know they're on a real long waiting list and actually their problem can be solved with that with wiser um you know a large language model version of a therapist if that like person's um problem is sold 80 of the time but actually 20 of the time it isn't and they still need a mental health professional that have human at the end of it um you've you've like diminished the demand on the service by 80 and so i think we need to start thinking about it like that rather than like we can't serve all people with ai is more like can we reduce the demand and pressure on the service with AI and actually expose the people that really need the help from a human to that 20%.

31:32And actually we do have the demand in the mental health, you know, professional workforce to support those people. That's an amazing win. Like that's when technology and plus human is an amazing thing. I'm forgetting the second part of your question, but it was something around sushi, bad sushi. Right. Yeah, it's very helpful and actually made before the second part of the question, it made me think, even in the work that we do, I think it mirrors what we're seeing, which is we classify every point of patient interaction, again, around like the admin side, booking, billing, questions, support, et cetera.

32:09And what we find is that basically it falls into one of three categories. One category is entirely service, non-healthcare-based correspondence. Nothing to do with healthcare, might as well be returning shoes online. That's one chunk of correspondence. Another chunk of correspondence is stuff that relates to operating procedures of the provider. So how much does something cost? When do you get a follow-up? So then there's this like clinic specific thing. And then there's everything that actually connects to the booking and the payment cycle. That's more like data rich. And it's interesting to hear what you say, which is like the 80 % who are happy to do it at 12 a.m.

32:45when they can't afford it or have a long wait list and 20 % don't. We often see something quite similar, which is actually responding to an entirely non-clinical service-based question. can still take you six minutes. And it's the same six minutes that it takes you to respond to a clinical and service-based question. So like, where do you want to spend those six minutes? Do you want to spend the six minutes saying, it was our absolute pleasure. Thank you so much. Do you want to spend six minutes there? Actually, would you rather have 12 minutes to spend on the clinical question? Probably the 12 minutes on the clinical question, because the other part doesn't really matter as much.

33:15So I think it matches in sort of some of the stuff that we see. yes the second part of the question was sushi um at some point you use the example earlier at some point you eat bad sushi and i've been in that position as as others have as well and like it takes you a while to eat that to eat sushi again like you know what i'm i'm good for now i'm i need six months have we had has ai had its bad sushi moment yet or is that or are we gonna are we gonna skirt around it or is this like you know do you think it's an unavoidable truck that's coming toward us there's going to be a bad sushi moment what what's your sense i think our like level of acceptance with the qualities i think if you were if you're a regular claude user or touch bt user and you are would put back in the 2022 version of the first release of touch bt you would think like this is so slow and awful i'm never going to use this again this is bad sushi but actually at the time it was transformational sushi so like just to kind of pull that thread a little longer like I think our level our expectance of kind of what we expect from AI is going to increase and increase and like technology is going to kind of come with it so at the moment we're so used to kind of analog services so that when you know in the I'm in the UK at the moment and trying to access a prescription so I'm going on to the NHS app like that for me is an amazing service because before you had to queue at eight in the morning and you're at number 25 and they shut off the queue at you know 8 30 so if you don't you're not like number two in the queue you don't actually access your gp like that was the service that we're used to so anything is better than that so i think like we should be in the in the nhs we came up with a strategy called digitize connect transform and it was impossible to think about ai when 100 of our healthcare service was on paper so we had to spend a lot of money a lot of time on digitizing service which i know is like an obvious point but it was like don't try and do the ai tools just focus on getting people on you know ehrs and then once everyone's on ehrs then start working on interoperability do the connect piece like how does the different care you know social care primary care secondary care whatever how does the ehr kind of follow you around so in the nhs we have an nhs number a single patient identifier that follows you from cradle to grave i know that that is a luxury in the us but um you know a lot of great work is happening there um and then the third piece is the transform piece like once we've done those two really boring steps very expensive very timely um then we can start to like layer on the transform transformational services but and then the kind of quality of those services get better and better and better with time so i think the sushi like the the the quality of sushi that we have now might be blowing our minds but we should be expecting the sushi to get better and better and better over time and yes we might have a terrible time um with ai at the moment maybe like there's a hallucination or there's like a dysfunction of some sort if we're using a chatbot or maybe they miss a diagnosis even worse or maybe they just like miss you out altogether because there's been some kind of error like that is the equivalent of an error like a you know food poisoning um and i hope that the like error is not so bad that actually it like removes that person altogether from ever experiencing or wanting to experience ai again so we're seeing this a lot at the moment in the uk people are removing their records from um data platforms because they don't trust the service or technology um because of some bad experience that they've had so that is that should be an available option for them but i think we should like really really radically prioritize the quality of service people receive so that they don't want to like remove themselves from that service because the technology is only going to get better.

37:11The services should be getting better with that technology. And so, yeah, our level of kind of expectation of that quality should be getting better and better as well. Liz, you've been so generous with your time. So thank you. I guess a closing question for you. and I think you sort of push on it now is this idea of in 2022, it blew our minds when you could, you know, type in whatever you wanted and get an answer, even if it took a minute to get that answer. And if it was right 50 % of the time, it was still better than what you had before. So there is definitely this feeling. And even for ourselves, you know, we're hands-on building around AI every day, building AI into our product.

37:54And every now and then we get this feeling where we say so we ain't seen nothing yet it's this this thing is still we're just in the early innings it's going to go it's going to get better and so i asked this question of a previous guest on the on the show and it was the wrong question to ask and they called me out on it they said that's a terrible question i said you know where are we going to be in five years like that's a crazy question you know five years is you know we may not even be here as a as a species at this rate in five years so where will we be in in in five months where we'll be in 12 months what what what what is the next inflection point in healthcare, in AI, whether it comes in five months, 12 months, whenever it is, where do you think the sort of the next gear shift is going to come and what do you think it's going to look like?

38:36What will be the defining trend of the next gear shift in whatever time period that is, be it five months, or I still think it'll be a bit longer than that, but needless to say, it's coming.

38:49I like to think that healthcare is becoming more kind of user experience led so the the kind of consumerist um user experience that we're used to in other sectors like financial services we're able to access our whole digital banking records we're able to like move money around very freely we're able to kind of get money back if there's like been a transaction that we don't recognize you know like it's pretty for me i'm very kind of happy with the digital banking service that i receive i'm not happy with the healthcare service i receive especially when I'm in the US and so I really hope that like the experience of the patient becomes more and more prioritized as we become more armed with the information like we have way more information about our records the services we receive you know the models that are being used on us and I hope that that kind of triggers more kind of user-centric experience for the patient I don't know about like what the big shift is going to be i like to think that like i think the foundation models have been quite quiet on um health care like we know you know the leads of health care and open ai and anthropic and deep mind like i have no idea what they're planning but i imagine that like if i'm a health system and i'm looking at like the 100 ai models that are deployed in my health system you know if it's a radiology algorithm here in the radiology department if it's a you know patient engagement chatbot on the patient platform place i can imagine that over time foundation models can be kind of eating up those verticals a little bit and we'll start to see like a platform way of you know consumers they might like buy up some of those um use cases or they might kind of integrate it better into a platform but i can imagine that there will be less of these like hundreds of pilots going on and more like um you know platform based like on top of the hr or something um and i think that will be easier to govern you know easier to kind of hold to account like liability easier be like traceable um aspects of those foundation models but but crucially the patient will get a better experience because they're like integrated better into the workflows rather than having like one system for this, one system for that.

41:08So, yeah, I would like to think in the next year, patients will really feel the difference in terms of the experience of the care that they receive. But that's ever hopeful, ever the optimist. As I say, that would be great. I love that vision for the future. I'm very excited about that. I'm very excited about that in years. I'm very excited about it. Lucy, thank you so much. For everyone who's listening again, this is Lucy Or Ewing with CHI, thinking about AI, healthcare policy, bad sushi, and keeping things on track as it relates to what's happening right now in the technology and healthcare space.

41:47Lucy, thank you so much again for joining us. It's been really a pleasure to have you. Thanks so much for having me. It's been such a joy. Thank you so much for joining us on this episode of The Business of a Clinic. We hope you had a good time. We hope it was helpful. If you can think of anyone who should join the conversation, feel free to nominate them, send them our way. We'd love to invite them to participate. If you like what you saw, give us a like, share, comment. We hope we'll see you again next time. Thanks so much.