The Echo in the Machine

0:28Wait, you're listening. My mother's living room, watching the television with her. This is what we love in our reporting. They scour the earth far and wide. Oh, yeah. Going to unknown, exciting places like the shag-carpeted living room of my mother. No, and so we're sitting there, and, you know, my mother's hearing. It's not what it once was. And so, like most nights, she was watching with the closed captioning on. Oh, absolutely same. All right, right on. Anyhow, I think it was the local news. literally talking about things like filling up potholes. Okay. And as I'm sitting on the floor, sort of bored out of my gourd, I have one of those moments where a genuine question popped into my head, which was those closed captions on the screen, you know, how did those get there?

1:20Like, is there someone in Sandusky typing as fast as they can? Exactly right. Like, is it a human sitting in an office? Yeah. Or, and this was sort of my real question, like, is this one of those jobs that AI has already taken and replaced us? Okay. Okay. Did you have a hunch? I thought it was probably AI. Okay. Just based on my own real world experience. Based on life right now. Right. And I also thought then that, you know, like one quick question to chat GPT and we're going to get to the bottom of this. Right. Turns out that was not the case here. Hmm. As I looked into this, yeah, I found it was cacophonous in a way I didn't expect.

2:05I found ladies swearing at their televisions, students demanding to be heard. And maybe oddest of all, a whole chorus of voices offering us a path through the strange future we seem to be walking into. Okay. Hey. Roots News broadcast is being closed captioned. Subtitles called closed captioning. Access to television. It puts words into your world. Closed captioned. Closed captioned. See what's being said. So, Greg is signing. I will be speaking. Okay, just speaking. And not signing. Okay, I'm just thinking. Okay, you are good. Seems like the best place to start is, you know, all the way back at the beginning.

2:48One, two, three, four, five is fine. That's great. Okay. Great. With this guy? My name is Greg Leibach. And his interpreter. Brenda Kelly Fry, Certified Interpreter for the Deaf. Today, Greg is an attorney with a coif of silver hair, thin-rimmed glasses. He was born deaf in Queens, New York. Less than, I don't know, one and a half miles from the New York Met Stadium near the airport. And I come from a deaf family, my parents and two brothers and one sister. And growing up, he says, you know, during the daytime, he felt pretty darn integrated into the larger hearing world. I had neighbors who were hearing, and we all associated with each other, communicated with each other.

3:28We stayed outside all day until the dinner bell rang. But in the evening, you know, when all good Americans turned on their TVs, he was not. It was pointless to watch except for maybe football and baseball games, because there was basically no captioning. We'd have to look at the TV guide, and, you know, they had a symbol that said CC. News broadcasts, the occasional special, that was about it. And it very well might have stayed that way if it hadn't been for Greg. I guess you could say so. And so, fast forwarding, it is the spring of 1988 on the campus of Gallaudet University. The campus is beautiful and gated.

4:14The students are in lots of denim and oversized sweatshirts. I mean, it is your standard-looking college. with one exception. It was nearly 100 % deaf. In fact, it was basically the only four-year liberal arts college for deaf students in the world. This is disability rights attorney Karen Pelt Strauss. I was on the staff of Gallaudet's National Center for Law and Deafness at the time. She is fluent in sign language. And in 1988, on campus, she says, tensions were high because... The position for Gallaudet's president opened up. And until that time, Gallaudet, it had always had a hearing president.

4:54Yeah, in its 124-year history, all of them were hearing. Missed opportunity in the leadership department. Oh, absolutely. And so the students on the campus said, you have got to choose a deaf president. And the faculty said the same, and the staff said the same. And according to Greg... I was a junior. I was in my junior year. Who was actually the student body president as all this was going down. He and his classmates. We were very optimistic. Because of the three finalists for the job, two of them, Harvey Corsons and I, King Jordan, they were deaf. And the third candidate, a woman by the name of Elizabeth Zinser, not only was she not deaf, she didn't even know sign language.

5:37No. Okay. Okay. I mean, Zinser had no support on campus. And so, March 6th, 1988. Pat, students here behind me have been waiting all day outside the gates of Gallaudet University, while inside the board of trustees have been meeting, trying to pick a new president. We were all gathered in the gym, in the field house, waiting for them to make the announcement. And around, say, 7 o 'clock, it happened. We picked Dr. Elizabeth Ann Zinser as the seventh president of Gallaudet. No! Because she is a very talented educator. Oh, no. Oh, no. Yeah. They went with the hearing lady. Elizabeth Ann Zinser, she is the new president of Gallaudet University, Dr.

6:28Elizabeth Zinser. Who is neither deaf nor able to speak sign language. Why? Did they say why? Well, at least one of the explanations was pretty darn ugly. The university trustee's chairman defended the selection saying, deaf people are not ready to function in the hearing world. And the students, well, they go berserk. Why, why, here, why? We were all upset. Why, goddammit! I'm so damn angry that this makes me sick. Very upset. We've been here for a hundred, twenty-four years. We've just felt like somebody just slapped us in the face. Things were escalating. Everybody was on the streets. And escalating.

7:09I mean, people were throwing things. And escalating. Then I told them, stop. Do not damage. Do not vandalize anything. No violence, please. Because I knew that many people don't have the experience of seeing deaf people. We were sending the wrong impression. We're sending the wrong message. Sometimes, you know, the first impression is the lasting impression. So I didn't want the hearing people seeing us as a wild bunch of people. So, you know, we gathered at the front of the gate in front of campus, said, let's get organized. And that's when we started making plans. First things first, Greg and a couple of the others.

7:56We drove to buy a chain from the hardware store. We brought the chain back and we locked all of the gates. On campus. They hotwire some of the school buses and drive those in front of the gates. Blocking those entrances. Huh. So now it's really blocked off. Fatten down the hatches of the whole university. Yes. And in the morning. As the administration arrived. We don't want the university to open. We want a deaf president first. 99 acres was totally shut down. The students vowing to keep it that way until the board replaced Zinser. With a deaf president now. Deaf president now! That was it. Students succeeded in shutting down the school in peaceful protest.

8:46Picture folks on each other's shoulders, waving signs, banging drums. Almost immediately, faculty and staff like Karen joined the cause. We had a great time. It was a party. We marched around. We had different presentations. And we had doughnuts. And at least once, they pulled the fire alarm, which, you know, didn't bother the students, but bothered anyone who could hear. Oh, metal! That's awesome. By the end of the first day, Greg had become the official spokesperson. We have the president of the student body, Greg Leibach. And by the second day, media from all over the country had poured it.

9:25In their signing and in their faces, you see their convictions. PBS, ABC. Demonstrations. Demonstrations continue. I mean, this became a national news story, culminating with Greg appearing a nightline to debate the incoming president, Elizabeth Zinser. Really? Dr. Zinser, please go ahead. Thank you, Ted. As president of Gallaudet University, I want to indicate that the university is an extraordinary institution. It deserves to have the continuing strength into the future in its mission as an educational institution. Excuse me. Are you implying that a deaf person can continue that for the future?

10:04Not at all. Okay, so that's Greg. So Greg in the red-tied gray suit? Yeah, yeah, yeah. We've got like a split screen going on. There are captions on the bottom of the screen. And Greg, who you're about to hear again, talking through an interpreter, is on the right side of the split screen. If they haven't provided any deaf leaders, then obviously Gallaudet hasn't done a good job. If they have done a good job, then there should be a deaf president, someone qualified to do this. So intense. Yeah, it gets heated, like, as Zinsser tries to get going here again. What I'm really saying is that I do believe that deaf individuals have great capacities.

10:36He cuts her off again. I truly believe that a deaf individual one day will be the president. No, that's old news. I'm tired of that statement. One day, again and again. All right, folks, let me, excuse me one second. Let me ask. Okay, so this debate was captioned. Do you think that was, like, a special move? or was... Yeah, so this broadcast was actually open captioned, meaning that everybody who tuned in saw the captioning on the bottom of the screen. Okay. However, like that was not the case for the vast majority of the coverage of the Deaf President Now protests. And in fact, even the broadcasts that were closed captioned, like to receive those closed captions, to get them to show up on your screen, you needed to have one of these very expensive, clunky, decoders.

11:26So ABC is sending... Oh, like in your house. In your house, connected to your television. Think of it like a VCR, but it's a VCR that just allows your television to receive the closed captions. So very few people of just like the general American public would be seeing these captions. Oh yeah, like nobody. Yeah. Which like, it's so frustrating to think that was like the day-to-day norm for deaf folks at that time. But I mean, there's just something like particularly frustrating to imagine, like the folks who can't access a broadcast that is literally concerning their rights and their access, you know?

12:07Yeah. And I think that's probably part of why you see this sort of chain reaction of events coming out of this moment. So less than a week after the protest starts, Zinzer resigned and was quickly replaced by one of the deaf finalists, I. King Jordan. Everyone was just signing and jumping and cheering and screaming and everybody was so happy. But then you also have a whole bunch of laws get passed in the years following. This thing called the Decoder Act that required all televisions to have that closed captioning decoder built into it. A little thing called the Americans with Disabilities Act.

12:51And eventually the 1996 Telecommunications Act. And that bill basically is what brings captioning into living rooms everywhere. And the mandate is what? It's that by, like, the early 2000s, all new English language broadcast television had to be closed captioned. All, everything that goes out. With, like, very, very few exceptions, everything has to be captioned. And, I mean, Karen and Greg, they were central in pushing this requirement into the bill. Wow. Like, that is such a—like, go Greg. Like, go Greg, go Karen. I mean, that's a huge win. Yeah, yeah, yeah, yeah, yeah. And they say, like, it all sort of started at Gallaudet.

13:40That's absolutely correct. Once more, Karen Peltz stress. The protest introduced society to the way that deaf people communicate. They introduced society to captioning and sign language interpreters. And they impacted congressional votes.

14:01Yeah, so that's the why, right? Yeah. Like why we have all of these closed captions today. But the how, like how they were going to make all of these hours and hours and hours of those closed captions. Well, that's where this story gets just delightful, number one. And number two, I think starts to say a bit about what the future of access to information and media is going to look like for all of us. Okay. And we will get to that in a moment. But first. I mean, up to that point, live closed captioning had only ever been produced through highly trained, specialized stenographic shorthand. Imagine a court reporter with a strange keyboard.

14:51Just like fire fingers. Exactly. Okay. That is how captions are being made. So you've got dozens, perhaps hundreds of people sitting in offices with the television being pumped into their ears through headphones and they're just typing away at lightning speed. But by the beginning of 2003, it was becoming apparent that not enough steno writers were available to match the growing amount of content that needed to be captioned. This is Meredith Patterson. President at the National Captioning Institute. And back when she joined basically as an entry-level employee, she was handed this problem. Yes, at the very beginning.

15:33Okay, so you are there. You're this like junior member of staff. I was very junior, and maybe that's why I was tasked with experimenting with some software that we called the Black Box. Okay. It was basically a very simple early-day speech recognition technology. You know, like a speech transcriber. And her hope was that she could just take a live television feed, Good evening, everyone. I'm Kevin Christopher. I'm Nancy Cox. Plug it in and create the captions that way. However, For the scope of years, 18 doesn't just need to scale up. When she tried that, Don Blevins says you're in Jesus. It was inaccurate.

16:18It would miss a lot of content. Little things like the news broadcaster throwing to the weatherman would totally trip it up. It didn't include punctuation. And accents of any kind were an issue. However, what it could do pretty darn well was transcribe her voice, which led to a sort of crazy idea. Could you just like, could you do the thing that we're about to talk about? Could you do the thing that we are about to talk about? Okay, so let's try it a little bit faster. So let's try it a little bit faster. I won't be stopping so much. Where I won't be stopping so much. We're talking about the news.

16:58It's going to be a very interesting day with the news today. It is going to be a very interesting day today. What if she just echoed every word said on television into the computer? Maybe she could close caption that way. Period. it. Okay, yep, you can do it. Wow. Oh my god. She called it voice writing. Voice writing. Huh. That's a funny name for like being a parrot. Being a human parrot. Why is that so comic? It's just so funny. Oh, Lulu, we are just getting started here. So first things first, to see if this was even possible. I would sit in the back of the room during internal meetings. Picture just a sterile conference room with a drop ceiling.

17:46Trying to be innocuous, repeating everything they said. Everything. I practiced at home sometimes on just random newscasts or people on TV. And well, she got really, really good at this. Like that didn't mean that the captions were coming out really good, really well. As she started doing this, echoing into the computer over and over again, it would miss words or have trouble understanding her, her English, her voice. And so Meredith decided to meet the machine where it was at. She set out to learn to speak computer.

18:29And we are going to get to that. And we are going to get to that. Come on. Right after a quick break. Right after a quick break. Very odd.

18:53Stephanie, I'm going to call you right back to see if that fixes the echo, okay? Sure, sure, Simon, call me right back. Okay. Radio Lab Lulu here with Simon Adler, who is telling us a story about how student protests led to a mandate that closed captions be beamed through all of our screens. And we were just moving on to the wild, echoey way that captioners hoped to actually get them to us. That's right. Voice writing. And along with Meredith, who you heard before the break. Let's see. One, two, three, four, five. I don't know why that fixed it, but it did, yeah. Strange. Okay, okay. This lady right here, Stephanie Waverka, Director of Production at the National Captioning Institute, set out to figure this out.

19:39Okay, so here's my first question for you. Yes, sir. And I noticed this with Meredith as well. I think your voices have been forever changed by the work that you have done. There is a precision and a spacing that makes sure that not a single syllable goes by without the listener being able to catch what it was. Do you think I'm right? I think you are mostly right. Yes. Okay. Well, let me back up. When we began with this voice writing line of work, the computer software wanted to hear you sounding like a computer. What would that sound like? Can I get a demo? Absolutely. That would sound like something like this, something that is very articulate and also very robotic, hyphen sounding period.

20:34very quick, sometimes clipped. I hear you laughing. I know. This is how we spoke for hours and hours of our day. She says her vocabulary had to change as well. Yes, because there were certain difficult words for the software to distinguish. For example, in, an, and and. Like she'd say into the computer and, but it would hear an, or she'd say in and it would hear and. And so the workaround she found was to train the computer to hear a specific real word when she would say a totally made up word. Like a little code. Yes. So she, instead of saying the word in, I-N, she would say. Inly. Inly. Inly.

21:24I-N-L-Y. Inly. Which the computer would then hear and print on the screen as in. How did you go home at the end of the day and start talking like a normal person again? It could be difficult to speak like a normal person after leaving. This job really did change me. Because Inley was really, well, only just the beginning. I mean, once she figured out this hack, she began developing and deploying hundreds and hundreds of code words to work around the software's shortcomings. Homophones could be very difficult for the software. Two, two, and two, for example. The fix? Tuku for T-W-O, toodaloo for T-O-O.

22:09So if a sentence is, she has two daughters in college too, I would echo that as, she has two co-daughters in Lee College, comma, toodaloo, period. So that is... Wait, wait, wait, say that once more. Say that one, say it again. She has Tuku Daughters in Lee College, comma, toodaloo. I mean, it's a whole language that you then have to remember and follow. As Stephanie's brain melded further and further with her machine, she figured out she could trick it in other ways to make her life easier. So, for example. My fellow Americans. Back when George W. Bush was still in office, that's how he was referred to on the air.

22:53George W. period Bush. eight syllables, way too many to spit out over and over again. And so... I trained my software to print George W. Bush when I said GB. I accept your nomination. Hillary Clinton became Hilco. Barack Obama became Bombo. Rudy Giuliani at the time was Ruju. Question mark. That is too many syllables. Again, Meredith Patterson. I trained the system every time I said poof, it would print the question mark symbol. They learned they could trick it into not hearing and printing certain words, both the obvious ones. The software had a bit of a naughty side and would produce the most inappropriate choice when it had the ability to do so.

23:46And so I spent an entire day of work saying every profanity word you could come up with into the system. Programming them out. So if it hurt, it would just do nothing? Exactly. Oh, neat. And then there were some weird ones that they had to program out as well. The word garage. Because when you're captioning local news, a lot of things happen in the garage. The fire started in the garage. The man hid in the garage. But when Stephanie would echo the word garage into the computer. The software would, nearly without fail, print crotch. Creating some wonderful misunderstandings. A Moline couple has transformed their crotch into a haven for rock climbers.

24:33Hoping to address a community need that we weren't even aware of. And I mean, this thing, voice writing, well, it became the industry standard for closed captioning. I mean, if you ever saw a closed caption after 2003, it was probably put there through this technique. At our peak, we had over 150 voice writers. And that was across the country. We had a lot of people in California, aspiring actors. and they were probably captioning 400 to 500 hours a day. A day? A day, yeah. Meaning thousands and thousands of hours of television each week were accessible to the deaf, and thousands and thousands of hours of work were spent by these voice writers really forming relationships with their machines.

25:30I think the best voice writers really learned how to move with the software, almost like dancing.

25:45Because it wasn't enough to just tell the software how to respond to you. You needed to respond to it to really achieve the highest accuracies when you were on the air.

26:08So is this how we are still doing it? Are the captions going through this anonymous office building full of human parrots? Well, it's no longer really an office building because the pandemic has made a lot of this work remote now. And the pandemic changed more than just where the captioning was being done. When the pandemic hits, due to everything going online, due to all of the constant press conferences happening, there is once more a flood of stuff that needs to be closed captioned. And now they don't have enough voice writers to cover all of this stuff. And so they are once again in this position of, oh, man, how do we keep up?

26:55and by 2020, that technology they had started playing around with back in the early 2000s, just like the black box, AI running the feed directly into the computer, it works pretty damn well. It works well enough that you basically no longer need a human in the system at all. Meaning this dance, it's winding down. It's coming to an end.

27:25Today, Meredith says AI is doing around 50 % of the closed captioning. The National Captioning Institute is hired to do. Wow. And they haven't hired anyone to fill any roles that have become vacant in the last two years. Another human bites the dust. Yeah, I think it's tough because I as a person, I as a professional, am thinking and worrying a lot about how these new AI tools are going to impact me, my livelihood and my craft. And, well, this is in one way a story about a bunch of people being replaced by those sorts of tools. it's also a little bit of a story about how to use those same tools with a smile to like approach those tools with some excitement and with some creativity the tools that are that are replacing you they may eventually but like yeah why shouldn't you enjoy your time with the hand grenade before it goes off you know like okay wait wait wait but what okay what's your analogy here.

28:40Sure. I think what I'm trying to say is that our voice writers, they were trying to get their machine to produce accurate text. And of course, now we are asking AI to do all sorts of other things for us, from designing a drug to helping us process our feelings to making a picture to writing a song. But it can't do those things well without us. It needs us to help it, to play with it. Yeah. And I mean, well, it is so easy to just be down or scared or turned off by these new tools. Or opposed to them for running on stolen human work and guzzling energy. Sure, yes, that too. But I think regardless of how you feel about these tools, ethically, what these voice writers show is that back and forth, that dance, It can yield some very unexpected and world-changing results.