Episode #113, Agentic AI Special: "The Rise of AI Agents" - Joseph Connor, Founder of CarefulAI and Prof. at UCL

27:53I paint on a digital iPad and stuff. I created an agent that could paint like me, publish it on social media like me, access all the content that's on my phone and through proximity measuring, understand where I was and what I was likely to do next. And I gave it the agency to be able to publish images on my behalf in the language that I chose to use it. And frankly, it became pretty damn annoying. Why was it annoying? Because it was better than me. It was more productive than me. It was more eloquent. That's a triumph. That's not annoying, surely. Wait till you find yourself in the position that you've got the digital equivalent of you doing what you do, and you will find yourself thinking, God, I wish I could switch this off.

28:40And I was in the position to be able to switch it off. That's a good position to be in. But we're not all going to get that opportunity. So the upside for me is agentic systems run and managed by people. There is a technology world out there, very powerful people who believe that there isn't a role for humans in that process. There's a lot of what we can do to automate and remove people. But I see no value in that. unless you, I just don't see any value in it. But that's the humanistic type of my nature. I just don't think it's a wise option because you'll end up with conflict. If you can't switch it off, you can't control it.

29:16Humans are great at making life difficult for themselves. We have things called wars. And they'll make it difficult for technology if it's applied in an appropriate manner. Sorry, Joseph, that just can't happen though, can it? I mean, we have to be able to turn it off. control of this. We can't just let things go off and create businesses and do things online without a human layer, I would call it. You know what I mean? Without oversight, without governance, without things getting reined in. Are we really at risk of losing control of agents, do you think? Could that be possible? Let me draw an analogy not relating to agents, But I'm going to ask you a question.

30:02Do you have any control over what used to be Twitter, now called X? Would it serve you up in terms of content? I have influence. I wouldn't say control. Obviously, I can pick and choose who I follow and what my feed looks like, but I can't control it now. And do you have any control over Facebook and what it presents to you in terms of ads and the like? No, not total control, just a little bit of influence. So to make tech scalable, and I'm not hitting out at either of those organizations, just explaining what happens when you scale things up to a global level. When you scale things up to a global level, the person that you're engaging is a node.

30:41It isn't a person. It's something that you send things to and you're trying to understand the response to it. Right, yeah. And you can't give them the ability to have complete control over it because you can't scale your product. if you did that. You can imagine a world where you've got an agent on your phone that you'd have no control over, it just appeared one day. You start to speak to it, talk to it, give it all your information, ask it about things that are really important to your business. And it comes back with things that you think are important and you think are relevant, so you use it more.

31:16It learns more about how you do things. You may get to the point where it doesn't serve you the answers that you wanted anymore. Do you have any control of switching it off? Probably not. Oh, okay. I see where you're going with it now then. So if it is something like an X or Facebook, then of course I have no control over that at all. Do you have any control over how Claude or Anthropic, the other agentic systems that serve us with chat agents, do you have any control over how they operate? No, because it's not my platforms that I'm on. They own the platforms. So the point I'm trying to make is, if we live in a world where you're unlikely to have control, then the agents that we use day in, day out, we're unlikely to have control over.

31:57What are the big issues that you're going to likely to face in the future? Chances are that your interaction with it is going to be tokenized, deposited in a repository somewhere and your behavior and how you do your work and how you live your life and people like you is going to end in a product. It might be somebody who runs a digital firm might be digital. There may be the digital equivalent of you because the systems have learned how you think, how you behave, what your inputs and outputs are. Wouldn't it be good if you had some degree of control over that? Yeah. I think it would. So what is the answer then, Joseph?

32:35How would you get control over that? So currently, the answer is to create your own knowledge base, to have LLMs that you control interrogating that knowledge base and representing answers in a manner that you understand to your customer base that are not dependent on third-party systems. So having your own language model, your own database, your own interface, and it representing what you do in the manner that you want to do. So we have a product called Insight Scholar, and that's what it does. It enables you to share your insights in the manner that you want it to be shared, and that data does not leave your phone.

33:15Another issue is the tokenization of what we do. You and I are speaking, and currently that language gets put into a signal. That signal is encoded, goes to your PC, gets de-encoded, and it comes out of sound, right? So that encoding process exists in language models, tokenization. I think in the future, there's going to be a need for control over where that token goes. So I'm going to give you the name of my first child, which is really important to me. Her name is called Grace. Grace Conner. So that token goes to you. I want to make sure you don't do anything with that name that I don't want you to do.

33:55So you look after her name. Well, you can't. You can't do anything about it because it's just come over to you as a token. It sits as a token in the space between the both of us. Who owns that token? Whoever the provider of this radio studio is. In language model terms, which are GPT, Anthropic, and others, they've got lots of tokenizations of words that are important to us. They may be not members of the family. They may be intellectual property. They may be digital assets. They don't keep that word, but they'll keep a tokenization of that word to be able to surface and deliver services. I think the loss of IP, the loss of tokens, or knowing where those tokens go is going to be important to everybody because it's where your IP goes.

34:40Where does Grace's name go next? Where does your intellectual property of your digital asset management system go next? you want to know, don't you? Especially if you put chatbot front end to it, you want to know where that IP is going. So these are the things that are creating immutable tokens onto content, I think, is where we may end up unless somebody eats the hole of everybody's intellect very quickly. And there is a massive bunfight going on at the moment. Between the AI platforms, you mean? Yeah, there's a massive bunfight going on to see which conversations, intellectual conversations, can we capture, tokenize, and resurface as agents going on.

35:26Wow. And we don't know that, do we? We don't think about that. This episode was brought to you by Be Digital. Be Digital support leadership teams to optimize cost and get more out of technology investments. Be Digital and the team have unrivaled expertise with technology license management and data remediation and are therefore perfectly positioned to help prepare organizations for AI technology capability. And on the last point, BDigital have just developed a cutting edge AI readiness assessment, which provides tech leaders with a platform they need to make well-informed decisions about AI adoption strategy in 2024 and beyond.

36:10Go to BeDigitalUK.com to find out more and get in touch.

36:19Look, we sold, most people nowadays sold their soul to the internet 2.0 companies and are now being advertised, programmatic advertising as the result of that, you know? The Facebooks and the Googles of this world. This is just the next generation then of that, essentially. Incredibly more sophisticated, but it's nonetheless a similar premise, isn't it? Yeah, I think you have to be very careful about making claims to multinationals that you're in essence stealing into people's intellectual property. I think that would be inappropriate to do that. I think what we need to accept, however, is that when you speak to an AI agent, it's going to learn from what you do and it's going to tokenize your content so that it can give you a better service.

37:09I think what I would advise most people think about is, where is that content held? What is its use? So read the terms and conditions, the provider that you use, and see where the jurisdictions of those terms and conditions are associated. Right. And improve your understanding as to whether you actually own the IP that you impart in a two-party conversation with an agent, voice agent, or an agent that doesn't speak but does things on your behalf. Yeah, of course. So I'm a business manager of a typical business, okay, listening to this podcast. And I'm thinking, well, IP is a big part of our value proposition on our product, and I don't want it to go into the ether.

37:59And for whatever reason, I want to maintain control and I've got X amount of employees who I want to abide by that edict that I put out. What advice would you give to the business leader who has got concerns about this exact topic you're talking about, Joseph? I think it would be wise to recognize that everybody in your firm is using agentic systems because they are very, very useful. Oh, yes. You've got two options, well, three. You could bury your head in the sand and say, we're going to write a policy that nobody will have access to any agentic systems through the browser. And people are doing that.

38:39But they have very little controls about the bring your own device issues so people are using the mobile phones to create content. The most protectable position is to create your own. Give your internal staff what they want. If they're using agentic systems to draw insight, then build your own agentic system to draw insight and share it as you see fit. So have control over it. That's the better and more defensible position, I think, rather than saying, well, you can't use it, and it's not fit for purpose, because people are people. If they find things useful, they'll use them. Yeah, for sure. Absolutely.

39:16Now that makes sense. So I'm really keen to understand, what are you most fearful of, okay, in terms of society and, you know, humanity generally? That was a very big question, but there's a lot of wrong turns we could make in the next couple of years in relation to agendic AI and all the associated periphery technology innovation, okay? What are you most concerned about over the next, let's say, five-year time horizon? I suppose because I'm a people person, there's people not recognizing the challenges that they face. Technologists, makers will make. So whether using generative AI to make things for good or for harm, these things will get made.

39:58But who decides how they're used and how they're deployed? A person is still in that loop. So having people who understand the full impact of deploying AI beyond it delivers a service, I think is something that I'm most fearful about. I don't know if you've come across the Turing Institute. Turing Institute developed a very good framework for the education of people in the realms of AI. And what we did is we created it into a chatbot called Atlas. So it's a training and learning these systems. So you can engage Atlas and over six iterations, it will give you probably what you need to know for your position in that firm.

40:35And we make it freely available because it is one of our biggest concerns. We have no control in the way in which really, I mean, we're a relatively small firm on the edge of the Atlantic in this place called Wales. But it is... The epicenter of AI innovation, I'll have you know. Of course, yes. Sorry, I'm pitching us wrong. Silly me. But that's my biggest worry, is that people don't understand that they need to have a role. Everybody's got a role in AI. Yeah. Whether it's from a user, a designer, developer, deployer. And in Prado, we give people roles based on things that they can have control over.

41:17Yeah. So when we go back to a VC, we'll say, okay, this is the investment. What you need to do is have these people control the design, these people control the development, these people control the deployment, which these people are in some of the firms, some are in the supply chain. These are the dependencies. Manage it. Because if you don't, chances are your success may be out of your control. So I think all of us have got a role to play. But let's face it, not everybody's a technologist. We're technologists. Probably the people listening to this are technologists. Well, speak for yourself, Joseph.

41:50But yeah, I try to be. So we have a certain degree of comfort with things that we think we have control over and that we think is important in our life. Sure. But we've all got a role to play to make sure that things are fit for purpose. Absolutely. That fit for purpose question and how wrong can we afford to be are two questions that just don't get asked enough. How wrong can we afford to be, is this fit for purpose? Not now, but by the time whatever your product or your company comes into being or whatever you're doing exists. So same question, but what are you most excited about? What excites me is the opportunity to do things faster.

42:30And the doors that technology opens, particularly genetic systems for people who don't have the agency that they should have. There's going to be a leveling, I think. currently for example I've got a PhD in computing science, mechanical engineer MBA do I need that currently to operate is what I do, I probably do now I did 20 years ago in the future you won't need that we'll have systems that give you the degree of intelligence, that bright graduate in the office that you arguably can manage and get them to do things for you there are lots of people that don't have access to bright people that because of agentics in particular will have access to a degree of agency that they can wield absolutely so how they behave with that responsibility and that great power is exciting and worrying at the same time I was going to say I think there's a worrying element to that as well What I mean is the people who will have the foresight to have access to this capability will have such a massive advantage over everyone else.

43:49It potentially could drive inequality. Do you know what I mean? I think some people will jump ahead. Their productivity will just go through the roof. whereas people who are not benefiting from the power of agentic AI will obviously not. So how do we democratize it so that the capability is spread across a broader section of the population? Do you see what I mean? Is it going to benefit the few, not the many? We want it to benefit everyone, don't we? So how do we control that? Not everybody wants that. Not everybody wants that. We may want to, but everybody wants that. that. Look, it's going to be unequal initially.

44:36Of course it is like any other technology innovation that's ever existed. But we should have endeavor to want to make it a great equalizer and benefiting the many, not the few. And I think that's what governments will aim to try and drive, I suppose. Well, hopefully anyway. But I don't know. What do you think? So it depends what you mean by gigantic systems, but if we're just simply talking about things that enable you to reason and learn faster, they're going to be available on everybody's phone anyway. But most people currently don't use them. So I see in the next two years, people's awareness of their capabilities is increasing significantly.

45:16And the major providers of those systems, whether they be Microsoft, AWS, ChatGPT, OpenAI, Anthropic, et cetera, their knowledge base of everything that we do is going to increase significantly as well. So it'll create opportunities and disparities. These major systems providers will know a lot more about us. They'll know a lot more about how we think and how we reason than they hitherto would have been able to do. Is that going to create a problem for somebody down the line? I think it is. Unless these are your own systems, which are based on your own in-house, closed, non-internet access-based technologies.

46:00What new vocations do you think will likely pop up, Joseph? I mean, I was talking like an AI ethics is a new field that seems to be popping up at the moment. Do you envisage any others that maybe have not emerged yet or any existing fields or vocational fields that you expect to proliferate or become more popular? And also, do you think there's any jobs that really are a threat from a genetic AI? I think the jobs that will come up will be the management of AI systems at different levels, whether it be hardware, software, algorithmic, fitness of purpose. These are things that are already here, but managed by the few.

46:44But in different sectors, you'll end up with specialisms, whether it be agentic systems to produce film, TV, agentic systems to control reactors. They'll be all over the place. So new jobs will come up, but you will have to have a good understanding about processes. And I think where I see a missing gap in education is people's understanding about processes and people's understanding about how to be critical. So critical thinking. Because you recognize, because we both engage agents day in, day out, you become happy with their response but if you don't take it from a critical perspective it becomes problematic so that's where we created a podcast called critical ai on ai where you've got two you've got two language agents arguing with critical thinking over an important subject which is unbelievably impressive i have to say i couldn't i shared it around the office and we just couldn't believe that that was two agents talking to each other it's astonishing but it's so realistic this was obviously when I first listened to it anyway, I'm sure it's even better now.

47:52But we'll share that on the show notes. Congratulations on that achievement. It was very eye-opening. It was there to show the art of what is currently possible. Its primary purpose is to encourage people to think critically. Because how can you implement AI unless you think about things critically with an open mind about what its impact is likely to be? Yeah, for sure. Joseph, you're a professor in University College London. How is this getting regulated right now within universities and higher education institutions? Are they just accepting the use of LLMs for coursework and things like that?

48:30Or are you just emphasizing the quality of the prompts and looking at it that way, like with the calculator, when that became a part of the education process within maths in the 60s and 70s, or whenever it was. So yeah, how is that getting regulated right now, the use of AI within universities? What are you seeing? Okay, so I sit in a firm that specializes in regulatory markets, and universities is not a regulatory market. So I'll drop the word regulation and maybe put the word framework in place. So it's not legally required that anybody does anything in the use of generative AI in universities.

49:05However, most people who are concerned about education of our young are concerned about people's use of a genetic system. So how do you decide whether something's been generated with AI or not? So there is increasing move to platforms where you cannot access content from third parties. You can only put things that come from your head into your fingers into your hands in terms of answering questions. It doesn't mean you haven't learned about things through agentic systems before representing it. So there's that side of the coin. People are trying to put barriers in place. There are also people who are trying to say, okay, we can mark papers en masse using agentic systems.

49:48Are they doing that right now? Not in UCL. Right. Not in UCL, but I'm talking about... Yeah, for the possible. We're talking about... Let's remove the word UCL and education, put the word education in there. There are people who are trying to decrease the workload on teachers by putting agentic systems in place to manage the workload of marking. There has been, I think it was a paper quite recently, whereby with the full permission of the paper itself, a group of agents were created to think about a problem, an academic problem, to reason that problem, to research that problem, to write a paper on that problem, produce a paper on that problem, and it was accepted into a journal.

50:33So in certain quarters, there's a bun fight to be had about how do you create the best agentic system to help you improve your understanding of the subject matter. Yeah. But when we talk to kids, you might find something slightly different. So if we speak to children, as we do, about what their hopes and aspirations are of AI, most of them don't want it that we've come across. Really? They're fearful of it. Wow. They're really fearful of it. That was really surprising to me. Look at the opportunity that you got, the things that you could do. There's a generation of people much younger than ourselves say 15, 16 that value authenticity or are more concerned about this agenda that seems to be going around that AI is going to have your job.

51:18Right. Those people are concerned, children are concerned, why should we go into education and all this AI is going to do our job for us? So there is a narrative that's been created by people who are concerned about AI that AI is going to take over the world. and that's how the unexpected consequence of that is within people who are directly making the choices to what education they want to go in, why should I bother? I find that deeply worrying. Rather than having the conversation with people about, as in young people, about you've got a role in terms of making sure that this AI is important because you're thinking and the way you think and your values are important and it should influence it, you should be able to take control of it.

52:04That isn't happening. and the role of critical thinking in education isn't happening. People are just thinking, well, why should I do my A-levels or why should I do a degree? And all these systems are going to take it. What job have I? What role have I? That's dangerous, that mindset. It becomes a self-fulfilling prophecy if it comes true because if people don't care enough to want to increase their understanding and have their ability to reason and take in information and reason in lots of different subjects, What happens if somebody switches it off, this thing that's replaced them? What role are they?

52:44So I think it's incumbent on large technology providers to get their finger out and say, we will support education. Absolutely, 100%. Joseph, I've really enjoyed chatting to you. As ever, we could go on for another hour, I'm sure. We've talked about some incredible things and let's see how the next couple of years play out. But where can people find you? Where can they find out more about Careful AI? If you go onto the internet and look up carefulai, all one word,.com, you'll find us. If you want to find me on LinkedIn, it's Joseph Connor. Type in that and Careful AI, you'll find me. But I think what I'd like people to do is to go and have a chat with their own IT people and say, really, what are our IP issues?

53:25What's our plan for doing things around AI governance? And just sort your own issues out. because we are very, very, very busy trying to make things. I think there's a role for giving people recognising they've got agency for their own issues. So people like your own firm and the myriad of other people who are in this domain, we're educators, we are enablers, but people need to take action for their own systems. And I think if they don't act, don't be surprised if somebody eats your breakfast, is what I'd say. Absolutely, well said. and on that bombshell. Joseph, it's been a pleasure. Thank you so much for joining us on the Tech Leaders podcast.

54:06It's been a pleasure. Thank you so much, Gareth.

54:12That was great. That was fab. Obviously, as you probably gathered, I do know Joseph. We've had a few conversations in the past, but not like this, though. This was great. And there were so many things in this episode that I could highlight here. But I've got to go with my gut feeling, what is sort of at the forefront of my mind finishing this conversation was something that really blew me away, which was Joseph's take on how we'll react to having a digital avatar and a digital version of ourselves. Imagine an AI agent who can do our job perhaps better than we can. How do we respond to that? I don't know.

54:53Will we feel empowered? Will we feel insecure? Will we feel displaced? I honestly don't know how I'd react, let alone how other people would react or how society at large would react to that. But what's clear, I suppose, is this, is that most people have no idea what's coming. This technology is moving so fast. Innovation is going so quickly. The idea that we could create digital beings with the capability and perhaps even the agency to change the very fabric of the work that they're doing is huge. and it's very exciting, but it's also very unsettling. Yeah, so that was just a realization and just a really bizarre place to take yourself.

55:36But I think that's all I would say for now. That really stood out for me. But thank you so much for listening. If you enjoyed this conversation, please give us a like and a share. It really helps us. But thanks for listening. I really enjoyed this conversation. I hope that came across. Yeah, please give us a like and subscribe. It really helps us. Thank you.

56:00This episode was brought to you by BDigital. BDigital support leadership teams to optimize cost and get more out of technology investments. BDigital and the team have unrivaled expertise with technology license management and data remediation and are therefore perfectly positioned to help prepare organizations for AI technology capability. And on the last point, BDigital have developed a cutting-edge AI readiness assessment, which provides tech leaders with a platform they need to make well-informed decisions about AI adoption strategy in 2024 and beyond. Go to Be Digital UK to find out more and get in touch.