AI Avatar of Killed Man Testifies in Court

0:03Hello, and welcome to the 404 Media Podcast, where we bring you unparalleled access to hidden worlds both online and IRL. 404 Media is a journalist founded company and needs your support to subscribe. Go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co. I'm your host, Joseph, and with me are 404 Media co-founders, Sam Cole. Hey. Emmanuel Mayberg. Hello. And Jason Kebler. What was up? Um, let's go straight into one that you wrote, Jason, along with Matthew Golt, our regular contributor.

0:47The headline is, I love that AI judge moved by AI generated avatar of man killed in Rose rage incident. Let me just straight away play some of the audio and video. If you're watching on YouTube, um, of this avatar, it's just a few seconds, I think. Hello. Just to be clear for everyone seeing this, I am a version of Chris Pelkey recreated through AI that uses my picture and my voice profile. I was able to be digitally regenerated to share with you today. But here is insight into who I actually was in real life. Take a look. Then the video carries on and it shows actual footage of Chris with family members and friends, if I recall correctly.

1:39Jason, what did we just see in here? Yeah, yeah. And then at the end, let's play one more little bit where he basically forgives the man who killed him, which is very odd, or at least was very weird to watch. But yeah, this is Christopher Pelkey. He's a man in Arizona who was killed in a road rage incident by a man named Gabriel Horkasitas. and what you're hearing is the end, the very, very, very, very end of the trial where Horkasidas is being sentenced. And so basically, he had already been found guilty and this is a sentencing hearing where, as part of it, everyone is allowed to submit a statement, more or less, like a victim testimony, more or less.

2:33And so what this means is people who know Gabriel Horkasitas can, you know, submit like testimony saying this was a one-time incident, like he's a good family man, things like that. Those are sort of the other types of statements that were submitted. And then Christopher Pelkey's family was like, he was such an amazing person. We're very sad that he's dead, of course, like this is just a normal part of sentencing. And then the judge considers all of that and comes up with how long the person should be in prison. So... Happens every single day in cases all across the state. Very, very standard.

3:18And usually doesn't involve AI. Usually doesn't involve AI. And I guess what I'm saying is like, this AI was not cross-examined or anything like that. Like this is a pre-recorded statement, and we'll get into how it was made. But this was submitted to the court as essentially Chris Pelkey's quote-unquote own statement. So in this case, you had a man who was killed, who from beyond the grave submitted a statement about who he was as a person, and at the end, directly addresses the killer and says, I forgive you, which is... It's just very wild to think about. I think that this is very much... I think this trope is way overused, but this is very Black Mirror to me.

4:16This is the type of things that Black Mirror examined, especially early on in its run, where it's like, you'll be able to digitally recreate people after they die and you can continue to have a relationship with them. In this case, it was a man who was killed who was testifying at the trial for... I mean, it ended up being a manslaughter, but testifying at the trial for the man who killed him. Yeah. And maybe here, this is where we can just drop some more audio and video of that latter part of the video where, as you say, he sort of directly addresses his killer. to Gabriel Horkasitas, the man who shot me.

5:01It is a shame we encountered each other that day in those circumstances. In another life, we probably could have been friends. I believe in forgiveness and in God who forgives. I always have, and I still do. So how do we get... Well, actually, no. I was going to save this for later, but I almost just want to immediately address it. the family says this is almost like the victim impact statement from Christopher. That is not true, right? It is a creation that's been made, and we'll get into that in a second, by the family. And I'm, of course, not even talking about this family's grief or what they must be feeling.

5:47I just mean on almost a factual, cold, legal basis, it's not him talking. Right. I mean, what do you make of that? Yeah, I mean, I think that this case is a little bit, you know, it's really weird because you have a grieving family. And I don't want to downplay that at all. But you have a grieving family that's trying to figure out how to cope with this situation. And basically his sister, who we spoke to, a woman named Stacey Wales, she wrote this. Like this was actually not, the actual words were not generated by an LLM that was trained on Christopher Pelkey or anything. Like she wrote his words or quote unquote his words.

6:32And then her husband, so Christopher Pelkey's brother-in-law is the one who recreated his voice. So they use an LLM to train on his voice and recreate his voice. And then they also used Allura, which Emmanuel has talked about a lot, is basically a mechanism for recreating a person from an image. And so that is how this was made. Yeah. I mean, Emmanuel, could you maybe talk a little bit more about that tool just for a second? Is it an app? Is it a tool? Is it just free to use? Does it just require one image? What's the deal with it? I would really define it as a method most commonly used with stable diffusion, which is an open-weights AI model that can do all sorts of things.

7:29People know it mostly from its ability to generate images. And essentially, it's just a way of very quickly training stable diffusion to recreate the likeness of a person or an object or a scene or a style. It's a way to... Training stable diffusion took a lot of time and a lot of training data. And Allure is a way to use just a few images or inputs to make it do something very specific that it wouldn't be able to do otherwise. Gotcha. That makes sense. So very accessible, it sounds like. Basically, anybody could use it. And I presume that's why, you know, somebody like a member of the public or whoever may turn to it.

8:14Jason, what did, well, two things. Why did Christopher's sister want to make this? And then second of all, sort of what did they make of the result? Were they happy with it? Or so the why and how good was it? Yeah, I mean, it sounds like she did this because she was having trouble writing her own victim impact statement for the trial. I mean, I sort of can't imagine being in this situation and knowing the words to say. And also, I mean, thankfully, I've never been in that situation, but sort of knowing what to even ask for or how to feel about it. It should be noted that the family actually asked for the maximum sentence time because the family was able to submit what they wanted.

9:02But interestingly, Christopher Pelkey's AI statement said that he forgives the person who killed him. And he didn't specifically, and I say he, but obviously this video doesn't specifically ask for anything. but the tone of it is one that's sort of like it's very unfortunate this happened he says at one point uh you know in a different life we could have been friends and it's kind of hard to know like what to make of of the whole thing i guess and like truth wise like is that really what he would have thought like what do you mean well i can talk more about that i mean i think that there's There's no way of knowing what he would have thought because it's just, regardless of how well you know someone, I think that, you know, I have lost someone who was very, very close to me.

9:54And the first thing that happened after she died was all of these people who barely knew her started saying, oh, she would have loved this. She would have said that. She would have done this. and first of all, she wouldn't have done most of those things. I know for a fact, but also you can never know. You can never know what someone... And that's one of the really awful things about death. And I think that's actually... That's actually really important to this case because this is fiction. It is fiction masquerading as a legal document because you have no way of knowing what Chris Pelkey would have said or would have done because he is dead.

10:39And you just simply can't know that, no matter how close you were to him. And so you essentially have his sister who has written something for him. And really, it's masquerading. I mean, people have called it a puppet. The story went pretty viral, not just our version, but it was widely covered in the press. And you just can't know. Yeah, because some other people covered it because the videos came out, and I'm going to play another one in a second, and then we spoke to the sister and I got more detail. But you did mention framing this as a legal document, and it is literally. It's been entered into court, and this AI avatar has, in a legal sense, testified in this hearing.

11:31someone who should know about legal documents and the sorts that should be allowed and potentially shouldn't be allowed uh you know are judges and here is the judge's reaction to seeing this ai played um in their courtroom i love that ai thank you for that and as angry as you are and justifiably angry as the family is I heard the forgiveness and I know Mr. Hortasita appreciate it but so did I So the audio was not the best quality there but basically the judge says that I loved the AI and the family could have appreciated it but so did the judge Some of the coverage didn't really focus on the judge's reaction of course it's in the headline of ours because it's pretty important What did you make of the judge's reaction when you saw it, Jason?

12:32Yeah, I mean, I was really surprised that it was accepted into the court. And, you know, I'm not a lawyer, especially not a... Well, I'm not any kind of lawyer, but I have less familiarity with the criminal legal system than I do some of the other types of lawsuits that we cover all the time regarding lawsuits and privacy things. Yeah, copyright. But I know very little about the proceedings of a criminal court. But I have to imagine that judges have pretty wide leeway to determine what can be accepted and what cannot. So I was shocked personally that he accepted this statement and said that he liked it.

13:12And I think it really does raise questions about how well-informed judges, lawyers, victims, suspects, etc. are about how AI tools work. And we know that law firms are sort of foaming at the mouth to use AI tools because it can really cut down on the amount of time that it takes to do things, to go through really big legal documents. Sam has written several articles about lawyers who have written documents for the court that hallucinated other court cases because they used an LLM. And in some cases, the judges chastised the lawyers for doing so. And in this case, the judge didn't chastise the family for doing this.

14:11He said, thank you for doing this and accepted it. I mean, I think Sam can talk a little bit more about how the legal profession is using AI, but I don't know. I think it opens up a lot of questions. Like, can a victim's family call an AI chatbot trained on someone's text messages to testify in a court against a murder case? I have no idea. It raises all sorts of questions to me. Yeah, Sam, what do you think? On one side, we have the judge here embracing it, and then you've covered all of these cases where lawyers have basically effed up because they've been using AI in their own sort of writings.

15:00What do you make of it? I mean, I thought that was a big part of why this was such a strange and kind of shocking story. A lot of judges are threatening lawyers with like fines and fees and severe punishment for using LLMs in compiling documents for courts. It's like we see a couple of times, most recently, I think the MyPillow attorneys did this, but we've covered a couple of times where lawyers are compiling cases and they're citing other cases that are comparable. It's like, you know, see this case, that case to kind of hold up their own argument. And either the opposing side or the judge catches that the cases are just not real.

15:49They don't exist. that's because they're using AI. I don't know. Who knows which one? There's many. But for example, they're using ChatGPT. You can imagine them kind of typing in like, make some cases, like cite some cases for me as part of this case that I'm working on. And then they plug that straight into a court document and submit it. And that's really sloppy. And it would piss me off as someone who hired that lawyer. And it pisses off the judge because the judge is like, this is a waste of time. This is a mockery of the profession that you're in. So to see this judge in particular say, be so moved by it is just, I don't know.

16:32It's like, you went to law school, bro. Like, what are you talking about? This is purely the conjecture of this guy's sister. It's not anyone's words, but hers and her husband's and the family's. So it's just, it was amazing to me that the judge was so, I don't know, I don't want to say gullible, but like that's kind of what it is. And then they were asking for the full sentence. So like really was he so magnanimous as to forgive his killer? It should be noted that the judge gave the killer the maximum sentence as the family asked for. So it's like, it's pretty, the forgiveness from the AI avatar did not sway the judge to go more lenient.

17:21And it wasn't meant to. It wasn't meant to because it came from the family. And the sister told us, here's what she told us. She said, quote, our goal was to make the judge cry. Our goal was to bring Chris to life and to humanize him. And it's like, I think you can feel a lot of potentially different ways about it because I'm sure that any family that's been through this will say that they feel like they've lost their voice. Chris cannot speak for himself. He cannot speak to what happened that day. But that is just how life is. It's the definition of a murder. Or any case where someone dies, it's like they cannot describe what happened to them.

18:08I just want to note that there's a long history of people more likely to put their critical thinking aside where new technology meets death. And I think that's because there's such a longing and the unknown about death that people accept really wild ideas. It's something that I covered a few times when we were at Motherboard. So, for example, in 2015, when VR was all the rage, there were quite a few projects that claimed to let you connect with your dead loved ones by creating some sort of VR experience that recreate the likeness of a lost parent or something and having some voice acting in there and that letting you communicate with the dead.

19:03which again, obviously, is not true. There have been a lot of kind of Holocaust memorial projects that use holograms and other new technologies, which again are like well-intentioned, as I think the family is in this case. But as Jason said, you're not communicating with someone who has died. It is more like talking to a psychic, which I think, as a culture, we're more likely to acknowledge is a scam. But for some reason, when you put the label of technology on it, it is something that you can present seriously in a court. And with AI being the hot new technology, we've seen a ton of this stuff around death.

19:59I will remind people that Sam covered this company called Replica many times and that it is essentially an AI girlfriend app. It is a companionship AI model, but it grew out of this project where someone created an AI model of their dead partner, and that kind of had this huge glowing feature in The Verge. And again, it was seriously engaging with this idea that this allowed someone to talk to the dead. And then five years down the line, it's just like a sex robot of some kind. So yeah, this is a thing that continues to happen over and over again, but it is definitely not something you'd see admissible in court usually.

20:50usually yeah i mean you're absolutely right and i would just add that one of the core projects of ai in general is to recreate consciousness like that is the the goal of it when we talk about you know super intelligence and the singularity and things like that it's like to create a conscious being and we don't need to dive into it now and it's there's many reasons to be very skeptical of this but that's the entire uh selling point that's like behind all the ai safety and then there's a lot of you know transhumanists and like ai developers who specifically want themselves to be to live on as an ai they want their own consciousness to be like uploaded into the everlasting cloud and so i think that that is very often there like there's so many ai startups that that have this as as their goal and uh just to sort of like transcend death in some way and it and there's been many many projects like this and so yeah to see it in in a court i mean when i saw it i was i was shocked and it's like i'm very rarely shocked by anything with ai anymore but i'm like oh you've got to be kidding me like i cannot believe that this was accepted yeah all right maybe we'll leave that there uh when we come back after the break we're going to talk about our interactions with meta and i suppose silicon valley pr reps more broadly and specifically it's about meta's plans to put ai sorry facial recognition same sort of thing into its smart Ray-Bans.

22:35We'll be right back after this.

22:50For years, I've just been trimming my beard because I found shaving to be uncomfortable and razors to be expensive. Then I realized that I just didn't have the right tech. Staying well-groomed requires the finest and most advanced tools. Upgrade with Harry's. Get the best razors money can buy for a fraction of the price of bigger brands. Right now, you can get a$10 trial set for just$5 at harrys.com slash 404, which is exclusive to our listeners. Like I said, I haven't regularly shaved my face for years because I have sensitive skin and didn't like dealing with the fuss. Then I got Harry's trial set, which comes with a premium five-blade razor, foaming shave gel with aloe, which is great for sensitive skin, and a travel cover.

23:36I had the best shave I've had in years. Totally smooth and no razor bumps, which is huge for me. Harry's has the highest customer satisfaction in the shaving industry, a no-risk trial, and a convenient subscription model that you can cancel at any time. Don't like your shave? No worries. It's on them. Take your grooming out of the Stone Age. Get Harry's. Normally, their trial set is$10, but right now you can get it for just$5 at harrys.com slash 404. That's our exclusive link, harrys.com slash 404 for a$5 trial set. Starting your own business is hot this summer, but lots of us don't vacation alone and you shouldn't dive into starting a new business alone either.

24:19It goes a long way to have proven experts to help you navigate that great unknown. And sometimes it makes sense to have a tour guide. For millions of businesses, Shopify is their guide, a tool that simplifies and supercharges your business. Shopify is the commerce platform behind millions of businesses around the world and 10 % of all e-commerce in the US. It's also the tool we use for our own merch store. Setting up and managing our store couldn't have been easier thanks to Shopify's powerful user-friendly interface. Get the word out like you have a marketing team behind you. Easily create email and social media campaigns wherever your customers are scrolling, or strolling.

24:56And get started with your own design studio. With hundreds of ready-to-use templates, Shopify helps you build a beautiful online store to match your brand style. And best yet, Shopify is your commerce expert with world-class expertise in everything from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into with Shopify on your side. Sign up for your$1 per month trial and start selling today at shopify.com slash media. Go to shopify.com slash media. Shopify.com slash media. Summer is just around the corner and the folks at Mint Mobile have a hot take.

25:44Getting a summer bod is out and getting your savings bod is in. This spring and summer, we want skimpy wireless bills and fat wallets. And with premium wireless plans for just$15 a month, you can have both without breaking a sweat or the bank. Switching to Mint Mobile is a breeze. Soon you'll be getting great wireless coverage on vacation at the beach or at the lake. Say bye-bye to your overpriced wireless plans, jaw-dropping monthly bills and unexpected overages. Mint Mobile is here to rescue you. All plans come with high-speed data and unlimited talk and text delivered on the nation's largest 5G network.

26:21Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of premium wireless service for Mint Mobile for$15 a month. This year, skip breaking a sweat and breaking the bank. Get your summer savings and shop premium wireless plans at mintmobile.com slash 404media. That's mintmobile.com slash 404media. Upfront payment of$45 for three-month, five-gigabyte plan required, equivalent to$15 per month. New customer offer for first three months only, then full price plan options available. Taxes and fees extra.

27:01See Mint Mobile for details.

27:14Well, well, well, we're back. Well, well, well, Joseph, what is going on here? So Joe Rosary, the headline is, Well, well, well, meta to add facial recognition to glasses after all. Um, so if you're a, uh, long time reader of 404, you know what we're alluding to already. Um, but a while back, um, how long ago was it, Joe? Was it like maybe earlier this year or last year? I believe it was last October, October 2024. Okay. So yeah, um, in October, uh, these two Harvard students did something pretty wacky with the Meta's, they have these like Ray-Ban glasses like that you can wear and they have cameras in them.

28:05So these two Harvard students got a hold of these glasses and did something with them. So do you want to walk us through that original story first before we get into the smugness of it all? Yeah, so they take these Meta Ray-Ban smart glasses. I always change the order of those words every time I say them. but they took those and they have the camera on board and then they basically strung together all of these different technologies which would include PIMEyes which is the off-the-shelf facial recognition tool that basically anybody can use people have used it to identify January 6 riot rioters people have used it um to dox sex workers people have used it in all sorts of abusive and then somewhat interesting ways as well.

28:55But they paired basically PimEyes with these Ray-Ban glasses, and they called it eye x-ray, if I'm remembering correctly. But it would also go a step further in that PimEyes gives you the web page where that matching face was found online. So let's say you're looking at somebody with the glasses, it captures their face it then puts their face through pym eyes and it brings up let's say the website of the law firm that this person works at then you know we could take the name attached to that put that through an llm and look for more information and the idea was that it could basically dox somebody just by looking at them finding where they live finding where where they work all of this other stuff and puts that all together into one package and they made this video uh which went viral uh on twitter slash x of the two students using it they said on dozens of people without their knowledge like unsuspecting and i would say victims you know maybe some people were okay with it.

30:05But just going around and revealing the identities of people who were anonymous members of the public. And this is, you know, the big, big fear when it comes to putting not just facial recognition tech in the hands of anybody, but facial recognition tech in the hands of anybody in a surreptitious manner in that I can just look at somebody and my Google Glass from a million years ago. Well, now my meta Ray-Bans are going to reveal to me who that person is, where ordinarily they would be anonymous. And if you walk down the street, I'm never going to see them again. Now I know who they are and I know everything about them.

30:43And I mean, it's a terrifying proof of concept. So we covered that at the time and we'll get into more detail in a second, but meta pushed back a lot against that article and they didn't really like that we were reporting on it the wording that you used in the story that we're talking about today is they uh let me get it right they chewed my ass off yes um that's a technical term yeah so with that context in mind what is meta announcing now as of uh last week yeah so there's a report in the information that meta is actually working on facial recognition tech for its meta ray bands and like we don't know all that much but basically they're working on it actively to put that capability into the glasses and they're also doing or they're also exploring this other thing which is not having or potentially not having the light turn on on the glasses.

31:48So usually when you put the meta Ray-Bans into recording mode or whatever you want to call it, a little light will appear, which will indicate to other people, oh, hey, that person's recording. I don't think enough people wear these glasses for that to be common knowledge, to be perfectly honest. Like, you know, if someone's shoving an iPhone in your face or a GoPro in your face, you don't know that the little red light on the pair of Ray-Bans means you're being filmed. Regardless, Meta is reportedly looking at getting rid of that as well. So, after all that coverage we did to the students and Meta's, you know, little meltdown about it, well, well, well, they're putting facial recognition into the glasses themselves anyway.