The Sound Barrier #1: The myth of hearing

10:45Matthew Winn:I was sort of half asleep and I was imagining notes jumping around in space. And by the next morning, they had sort of crystallized into what I named the scale illusion.

11:01Diana Deutsch:The scale illusion.

11:07Diana Deutsch:Just like before, this illusion consists of two tone sequences, one in each ear.

11:15Matthew Winn:So there's one channel alone.

11:17Diana Deutsch:Some high notes, some low notes.

11:19Matthew Winn:And then the other channel alone.

11:22Diana Deutsch:Some more high notes, some more low notes.

11:24Matthew Winn:And then you hear them together again.

11:33Diana Deutsch:If you're listening on headphones, you're probably hearing all the high notes on one side and all the low notes on the other, even though those notes are actually jumping from left to right. That's your brain editing the sounds. It's separating them to reflect the way the world usually is.

11:50Matthew Winn:In the real world, one would assume that sounds that are in a higher pitch range are coming from one source and sounds in a lower pitch range are coming from another source.

12:02Diana Deutsch:So that's what the brain assumes is happening here.

12:05Matthew Winn:The brain reorganizes the sounds in space in accordance with this interpretation.

12:13Diana Deutsch:Just like removing echoes, this kind of brain editing would normally help you make sense of the world. But Diana's illusion is explicitly designed to fool the brain into making a wrong guess. And not everyone's brain makes the same guess.

12:28Matthew Winn:Left-handers as a group are likely to be hearing something different from right-handers as a group.

12:34Diana Deutsch:Right-handers tend to hear high tones on the right side, but for left-handers, it's more complicated. They're likelier than other people to hear high tones on the left or in even weirder ways. All of this reorganization, the way the brain edits our hearing to help us navigate the real world, it's sometimes called top-down processing.

12:54Matthew Winn:Top-down processing occurs when the brain uses expectation, experience, and also various principles of perceptual organization to influence what is perceived.

13:12Diana Deutsch:Instead of bottom-up processing, which is sensing the world and then having that travel up to the brain, top-down processing means that our brain is influencing how we hear.

13:23Matthew Winn:To some extent, our brain is hearing what we are expecting to hear.

13:28Diana Deutsch:In a sense, a lot of what we perceive isn't actually us hearing sound waves hit our eardrum. It's a prediction of what those waves should be. To illustrate this, Diana uses something called the mysterious melody.

13:42Matthew Winn:This is a well-known tune, but the notes are presented in different octaves.

13:48Diana Deutsch:For all the non-music folks out there, an octave is basically a standard range of musical notes. In this illusion, the notes stay the same, but which range they're played in changes. So instead of playing do-re-mi in the same range with all the notes next to each other, you could play do-re-mi with the notes jumping into a different range. So Diana takes a well-known tune, doesn't change the melody, just changes the range.

14:18Matthew Winn:And the question is, can people recognize this melody?

14:28Matthew Winn:And in fact, people can't recognize the melody.

14:32Diana Deutsch:Now listen to a simplified version of the same sequence.

14:35Matthew Winn:In this case, all the notes are in the same octave.

14:39Diana Deutsch:Same range.

14:48Diana Deutsch:You know what it is.

14:49Matthew Winn:Yeah, indeed, it's Yankee Doodle.

14:50Diana Deutsch:And a lot of times when people go back and listen to the scrambled version, they can hear Yankee Doodle in there.

15:06Diana Deutsch:When you have a frame of reference for what you're hearing, when you have an expectation, it actually changes what you're hearing. Illusions like this tend to circulate around the internet every once in a while. Like this one where depending on which word you're thinking of, you might be able to hear either Laurel or Yanny. Laurel. Laurel. Remember last year when that Laurel versus Yanny thing, everybody's going nuts over? Well, there's a kiddie version of it making the rounds right now. This is from Jimmy Kimmel's show. And he starts by pulling up a clip from Sesame Street, of all places.

15:40Michael Chorost:I move it to follow you. Move the camera. Yes, yes. That sounds like an excellent idea. All right.

15:47Diana Deutsch:And pay attention to this, because tell me if you hear Grover say one of two things. That sounds like an excellent idea, or that's a effing excellent idea.

15:55Michael Chorost:Are you ready? Okay.

15:56Diana Deutsch:Yes, yes, that sounds like an excellent idea.

16:01Michael Chorost:Kim, well, what did you hear?

16:02Diana Deutsch:It's a f***ing excellent idea. You heard that? Yes, I did, yeah. It's the first time I heard it. I didn't hear a curse word at all.

16:08Michael Chorost:And then the next 12 times I watched it, the F word was all I heard.

16:11Diana Deutsch:Just in case you want one more go at it, here's Grover maybe making a lot of parents upset. Yes, yes, that sounds like an excellent idea. This type of misperception is true to an extent with all our senses. We've all seen visual illusions, or you might remember the debate around the dress. But Diana eventually found that the various ways our brain edits the world, they're not just due to hard-coded differences, like whether you're right or left-handed. Brain editing can vary from person to person based on life experience. To prove this, she asked listeners to determine whether a pattern is going up or going down.

16:51Diana Deutsch:For people who know a bit of music theory, this interval is a tritone, which is exactly half of an octave. So to get from note to note, you travel the same distance whether you're going up or down. If you don't know that much about music, all you need to know is that this is a particularly ambiguous pattern. But Diana does something really interesting in her experiment here. She plays the melody in a bunch of registers at the same time. So you might have an extra hard time figuring out if it's rising or falling.

17:22Matthew Winn:And sure enough, you get huge differences from one individual to the other. And this is something that really does surprise people.

17:33Diana Deutsch:I hear it going up, and Diana found that other people hear it going up. but some people hear it going down. What's truly mind-boggling is that Diana's found that the difference in how two people perceive this pattern, it might come down to where you grew up. Believe it or not, when Diana compared two groups, people from southern England and people from California, she found that the English people tended to hear this pattern as rising, whereas the Californians heard that same pattern as falling. Diana's hypothesis is that based on where you grow up, you tend to hear different pitches as low or high.

18:10Matthew Winn:It has to do with the pitch range of the speech to which you have been most frequently exposed, particularly in childhood.

18:18Diana Deutsch:So if you hear that first pattern, which goes from the notes D to G sharp as falling, you probably hear this second pattern, which goes the exact same distance from the notes A to D sharp as rising. Or vice versa. But ultimately, the mechanics of all this are still pretty much a mystery. Scientists don't really know how all this brain editing happens.

18:43Matthew Winn:I mean, we know that the brain does that, but we don't really know how.

18:49Diana Deutsch:In a sense, it's almost like we're all listening to a play performed in our heads just for us. There's a script, the entire world of pressure waves bouncing around. but how we actually hear it all is up to the performers. In so many ways, our brain dictates how we hear the world. But even though we don't know exactly how our brain does this, there are times when harnessing that brain magic starts to become a lot more important. It was like my hearing was pouring out of my head, like water out of a cracked jar. Coming up after the break, one man's quest to hear his favorite piece of music again.

19:27Diana Deutsch:That's next.

19:40Michael Chorost:Support for this show comes from Shopify. Starting a business is not easy. It can come with a lot of self-doubt and questions like, what if I fail? Or what if nobody buys what I have to sell? But if you are a small business owner pushing through that self-doubt, Shopify wants you to know that it is there to help along the way. Shopify is the commerce platform behind millions of businesses around the world. And according to their numbers, 10 % of all e-commerce in the U.S. From household names like Magic Spoon to brands just getting started. Their design studio lets you build a beautiful online store that matches your brand style.

20:18Michael Chorost:They offer a host of helpful AI tools, plus built-in marketing tools to help you create effective email and social media campaigns. And if you ever get stuck, their award-winning 24-7 customer support has your back. It is time to turn those what-ifs into Shopify today. Sign up for your$1 per month trial today at shopify.com slash unexplainable. Go to shopify.com slash unexplainable. That is shopify.com slash unexplainable.

20:54Michael Chorost:Support for the show comes from Quince. The other day I was out shopping, picked up a cute dress, but I wasn't totally sure it was my style because I've been kind of moving away from sleeveless stuff lately. Then I brought it home and tried it on over a little lightweight black T-shirt that I got from Quince. And I am literally wearing the whole outfit right now as I record this ad. The shirt from Quince was just the perfect thing to pair with the dress to give it sleeves. And the shirt is light and breathable enough that I'm not like sitting in the studio too warm. So as a person who loves to layer clothing, I really, really appreciate that Quince has a wide array of shirts and a whole bunch of different cuts.

21:41Michael Chorost:Lots of options for people like me. But if you are looking for not just shirts, but a whole range of lighter, cooler wardrobe pieces as part of a summer refresh, Quince also has you covered. Quince can help you find some new, breathable outfits that are easy to wear and still look put together. Elevate your summer wardrobe. Go to quince.com slash unexplainable for free shipping on your order and 365-day returns. Now available in Canada, too. So that is Q-U-I-N-C-E dot com slash unexplainable for free shipping and 365 day returns. Quince dot com slash unexplainable.

22:23Michael Chorost:Hey, everybody, it's Prime Video's Cassidy Hubberth and the Chicago Sky Skyler Dingens and I are bringing our new podcast, and mom to Chicago for Vox Media's The Lineup on Friday, July 24th. And mom is the conversation motherhood deserves. It's honest, unfiltered, and real. Skylar and I are diving into what it actually looks like to be both a working professional and a mother. The identity shifts, the career pivots, the mental load, and the stuff that usually stays behind closed doors. And at The Lineup, we're bringing that conversation live. It's a full day of programming that celebrates this amazing league and its players.

23:01Michael Chorost:Three stellar podcasts, four legendary hosts, one stage. Get your tickets for the lineup at voxmedia.com slash lineup. That's voxmedia.com slash lineup. See you all Friday, July 24th in Chicago.

23:19Michael Chorost:Move the camera. Yes, yes, that's an excellent idea. Yes.

23:24Diana Deutsch:Unexplainable, we're back. And we've been talking about the mysterious way our brain filters, edits, and even reconstructs the world that we hear. For some people, this kind of brain magic can be interesting to highlight as a party trick. But for others, it can be way more important. Okay, testing. One, two, three, testing. This is Mike Chorist. So it's like you take the word chorus, just add a T at the end. Mike's a science writer who was born with severe hearing loss, but he was able to use hearing aids. And starting from when he was 15, he became obsessed with Bolero, the famous piece by Maurice Ravel.

24:04Michael Chorost:It was this riotous melange, which such a fascinating drumbeat underneath it all really thrilled me and fascinated me.

24:16Diana Deutsch:He particularly loved the way the melody would gradually evolve over the course of the piece.

24:21Michael Chorost:Each repetition is on a higher level. It's louder. The resonance is deeper until it reaches a climax. So it's a very auditorily overwhelming piece of music.

24:36Michael Chorost:He would listen to bolero over and over and over. It was kind of my piece of music that I would come to again and again and again to test out new hearing aids. So it's always been an auditory touchstone for me.

24:56Diana Deutsch:And then one day in 2001, the limited hearing he still had started disappearing.

25:03Michael Chorost:I was standing outside and I rent a car and I suddenly thought that my batteries had died, my hearing aid batteries.

25:11Diana Deutsch:Suddenly, the traffic on a nearby highway started sounding different.

25:15Michael Chorost:It was just that sound that you associate with cars going by. You know, vroom, vroom, vroom. But all of a sudden, it sounded more like... As if somebody had dumped a whole bunch of cotton onto the highway.

25:32Diana Deutsch:Pretty soon, Mike found out he was quickly losing what was left of his hearing.

25:36Michael Chorost:It was like my hearing was pouring out of my head, like water out of a cracked jar. So after about four hours after that initial realization, I was essentially completely deaf. It was just such a shocking experience.

25:53Diana Deutsch:But Mike was eligible to receive a cochlear implant. It's a surgically implanted device that can offer a form of hearing in some deaf people. Many people in the deaf community prefer to communicate using sign language or lip reading rather than using a cochlear implant. But for some people, especially people who've lost their hearing later in life and want to continue using their native spoken language, cochlear implants can be helpful tools. The cochlea is this tiny spiral-shaped organ inside your head. And a cochlear implant is a string of electrodes that's carefully inserted inside that spiral organ.

26:29Diana Deutsch:This is Matthew again, the audiologist who actually works with cochlear implant users to help them understand their experience. There's this external part that looks like a hearing aid, but is not a hearing aid. It's a microphone and a computer that analyzes the sound and sends instructions to those electrodes that are inside the ear. The implant essentially bypasses a lot of the ear. It directly activates the cochlea, which then passes an electric signal onto the brain. But cochlear implants don't just reproduce normal hearing. Mike says that reducing sound to digital ones and zeros and beaming them directly into your brain, it can sound strange.

27:07Diana Deutsch:It was shocking. It was not at all what I expected. When Mike's implant was turned on, the first thing he did was listen to his own voice.

27:15Michael Chorost:And my voice sounded really weirdly high-pitched. I almost sounded like... Yeah, it was that kind of sound. It was like listening to a Demented Mouse.

27:28Diana Deutsch:Matthew actually gave me a program he uses as an audiologist to simulate various types of cochlear implant sounds. So here's a general idea of what it might have sounded like to Mike.

27:44Michael Chorost:It was very upsetting. I thought the world would sound pretty much like I heard with hearing aids, just fuzzier. I was completely unprepared for the huge difference in pitches.

27:55Diana Deutsch:Because of the way the implants are designed, they tend to make everything seem a bit high-pitched.

28:01Michael Chorost:So when you send a signal to any part of the cochlear implant, the brain will interpret that as a high-pitched sound, even if it's a low-pitched. Which is why everything can sound all mousy. But the interesting thing is, within just a day or two, I started to hear low-pitched again. And part of that was my brain adapting to it. My brain was saying, okay, this is my voice. I know it's supposed to be a low pitch. However, right now I'm hearing it as a high pitch. Never mind that. Because I know it's a low pitch, I'm going to interpret it as a low pitch.

28:34Diana Deutsch:Essentially, Mike's brain was editing the world for him.

28:39Michael Chorost:So very quickly, my brain started figuring out, okay, the world sounds really weird, but I'm going to try to fit that into my preconception into what the world is supposed to sound like. He was taking command of his own top-down processing. So within hours, I stopped sounding like Mickey Mouse to myself. And then Mike started training. I got the audio books of the Winnie the Pooh books. And I remember the first time I put the tape into the cassette player, I played Winnie the Pooh and some bees. I think that's the one. I couldn't make it out at all. It was just complete gibberish. But he also had the physical book, so he read along with the tape.

29:24Michael Chorost:So I was able to start matching up the weird input that I was getting with the words on the page that told me what that input meant. What about a story? Said Christopher Robin. Could you very sweetly tell Winnie the Pooh one?

29:44Michael Chorost:This is what the S sounds like. This is what the phoneme poo sounds like. We knew the... So it is a process of remapping.

29:54Diana Deutsch:According to Matthew, this process of brain remapping is a pretty normal experience for cochlear implant users. Any good audiologist would say to someone if they're thinking about a cochlear implant that when you first get it and it first is activated, you probably won't understand much at all. But over the first six months, maybe the first year, your brain learns to reorganize how it associates sound with meaning. Training's more accessible these days. It's certainly not as DIY as it was for Mike 20 years ago. But this kind of improvement can still be hard to believe. A lot of the people that I've worked with will say, now when I listen to my spouse, it sounds like her voice.

30:36Diana Deutsch:which baffles all of us who work in this field because if you look at how the ear is being activated, there's no explanation. I mean, not to be too on the nose, but it's unexplainable, right? So there's no way that that could possibly be true. And yet a lot of people say it. Tweaking settings on the implant does make it work better, but that doesn't account for most of this incredible improvement. A lot of the success of the cochlear implant is really a testament to how strong the brain is working, rather than a reflection of the high quality of the sound input. Our brains have an almost uncanny ability to predict language and fill in gaps, even when we hear something muffled or distorted.

31:21Diana Deutsch:But while cochlear implants work pretty well for speech, they don't work nearly as well for music. Music is just a much more complicated kind of sound. You need to distinguish melodies and harmonies and textures, and most fundamentally, pitches. And an implant only has a small number of electrodes. You have to simplify all the frequencies, and you can think of it as like pixelating the sound. Making this even harder, because the cochlea is filled with fluid, it's hard to use electrical pulses to stimulate the exact part that codes for the right frequency. Instead, the pulses kind of spread out around the part that codes for that frequency.

31:59Diana Deutsch:Let me make an analogy. Suppose you're playing a note on the piano. You can be really careful and hit the exact key you want, or you can be kind of crude and put your whole hand down on the piano. Like, you're going to be in the right ballpark of the note, but you're not going to hit the exact note very clearly. So a cochlear implant is more like putting your whole hand down on the note. It's not a very precise frequency you're hearing. When you take all of this into account, Translating music with a cochlear implant can seem almost impossible. The current design of cochlear implants isn't set up really for music.

32:34It's set up to understand speech.

32:37Michael Chorost:But I wanted my bolero back.

32:50Diana Deutsch:Even though Mike's brain had learned how to edit those high-pitched, tinny sounds to understand speech, Music still wasn't the same. It just sounded awful.

33:00Michael Chorost:Like, oh my God, you know. It was really shocking because, like, even if it gets twice as good as this, it's still going to be awful. Even if it gets three times as this, it's still going to be awful.

33:12Diana Deutsch:It was really bad. Mike upgraded the hardware of his cochlear implant. He upgraded the software. He even volunteered as a guinea pig for some tests on new equipment.

33:22Michael Chorost:So I would pronounce that as headphones. I hear the set of beeps and boops. I'm like, okay, which song is that?

33:33Michael Chorost:I'm like, I don't know. It's like, could anybody know?

33:40Michael Chorost:And for me, this was a very deeply frustrating kind of experiment because I know Twinkle, Twinkle, Little Star. I was like, that doesn't sound like Twinkle, Twinkle, Little Star to me. How could this sound like Twinkle, Twinkle, Little Star to anybody else?

33:57Diana Deutsch:Researchers I spoke to told me that some cochlear implant users just don't enjoy music that much. It's certainly harder to get used to than speech. And because patients are often told to focus more on improving listening to speech, music can get left by the wayside. But appreciating music through an implant can sometimes be presented as an insurmountable obstacle. You can see this in the movie The Sound of Metal, where a musician gets a cochlear implant after losing his hearing and then goes to this performance, listening to the song you're hearing right now. In this scene, the movie shows what other people at the performance hear, and then it gradually shifts perspectives to highlight what the main character hears through his cochlear implant.

34:47Diana Deutsch:The performance is so upsetting for the main character that he ultimately takes his processor off. He essentially decides not to use his implant anymore. You can find a lot of simulations online like this. So I asked Mike if these kind of simulations, or even ones like the simulations I created of a distorted voice or a distorted bolero for this episode, if they seem like accurate representations of what music sounds like through an implant.

35:16Michael Chorost:I think you have to be extremely careful when listening to these simulations because basically what those simulations are telling you is this is what the software is giving to the user. That's not the same thing as what the user hears. These are two very different things. When I listen to these simulations, and I have listened to them, it does sound a lot like what I heard on day one. It does not sound like what I hear in year 20.

35:49Diana Deutsch:For Mike, this was a combination of training himself with careful listening, but also tweaking the settings of the implant. Because with a lot of practice and effort and time, the experience of listening to music can improve.

36:03Michael Chorost:Yeah, I would listen to music over and over again. And I would try tweaking different settings. And I would go to my audio and I would say, these pictures sound really fuzzy to me. Can you do something about that? And so she would tweak how much electricity went to different electrodes. And so this was an iterative process that went on and is still going on.

36:28Diana Deutsch:After years of upgrades, tweaks, training, Mike's noticed some real improvement.

36:34Michael Chorost:But not for all music. Most of the piece of music that I enjoy is music that I heard with hearing aids. It's familiar to me.

36:43Diana Deutsch:Mike does listen to some new music, but preferring familiar music, it's a pattern that Matthew notices with his patients, too. And I think it's a testament to the brain filling in those gaps, conjuring the memory of what the sound quality should be. The implant sort of gives you just enough that the brain can put together the whole puzzle. And, of course, Mike is listening to Bolero again. Well, it sounds good.

37:08Michael Chorost:I really enjoy it. But there are things that I know that I'm missing. I know that I'm still not getting some of that intensity and the purity where the music is reaching for a crescendo in each of its iterations. So I know I'm missing that.

37:27Diana Deutsch:In a sense, Bolero is so familiar, it's almost like language for Mike.

37:32Michael Chorost:Bolero sounds really good to me because I know exactly what it's supposed to sound like.

37:36Diana Deutsch:This new Bolero is certainly different from the version he remembers. But Mike loves the new version.

37:43Michael Chorost:Even though the input I'm getting of Bolero is incomplete, and I can hear that it's incomplete, it is still a source of pleasure to me.

37:56Diana Deutsch:Ultimately, we don't really know exactly how our brain is able to do this. It can almost feel like magic. How it filters out echoes, how it shifts high tones to one ear and low tones to the other. how it can take a tinny, noisy input and rebuild a new version of Valera.

38:14Matthew Winn:We do this very complex calculation, but I don't think that we really know exactly how it's done.

38:23Diana Deutsch:Psychologist Diana Deutsch again.

38:25Matthew Winn:There are an awful lot of things about our hearing that we don't understand. And what we hear is often quite different from what in point of fact is being presented.

38:36Diana Deutsch:But we do know that the brain is constantly editing, shaping, and building the world that we hear. Our brain, our life experience, our familiarity with a piece of music, it all shapes how we hear. And what we hear. Which raises a pretty fundamental question.

38:54Matthew Winn:When an orchestra performs a symphony, what is the real music?

38:58Diana Deutsch:Is it in the mind of the composer?

39:00Matthew Winn:Or is it in the mind of the conductor who has worked long hours to shape the orchestral performance?

39:06Diana Deutsch:Is it in the mind of someone in the audience who's never heard it before and doesn't know what to expect?

39:11Matthew Winn:And the answer is surely that there's no one real version of the music, but many. And each one is shaped by the knowledge and expectations that listeners bring to their experiences.

39:23Diana Deutsch:The idea that to a very real extent, our brains conjure different individual realities inside our heads. On the one hand, it's a clear reminder to be humble. And not just for hearing. No matter how certain we are, what we perceive isn't unfiltered reality. So it's worth questioning ourselves at our most stubborn moments. At the same time, though, how cool are brains? I know they're this perfect reminder of our own subjectivity and humility. But I also just can't get over the fact that our brain puts on this fireworks show every day. and that a lot of people using a cochlear implant can tap into this almost magic ability to translate a few electrodes into this new, emotionally satisfying experience without scientists really knowing how the whole thing works.

40:15Diana Deutsch:There's so much we still don't understand about the brain and how it tries to make sense of the world, and it just makes me that much more excited for everything we're going to learn along the way.

40:44Diana Deutsch:This is just the first episode of four in our series, The Sound Barrier. On the next episode, a listener with tinnitus who heard this episode and got in touch to ask how to retrain her brain.

40:56Michael Chorost:I was thinking about the cochlear implant, like how they had to train themselves in a way, you know? That's why I was like, should I reach out?

41:04Diana Deutsch:That's next time. As for this episode, it was reported and produced by me, Noam Hassenfeld. I also wrote the music. It was edited by Catherine Wells, Brian Resnick, and Meredith Hodnott, who runs the show. Mixing and sound design from Christian Ayala with an ear from Afim Shapiro. Richard Sima checked the facts. Sally Helm and Joanna Salataroff gave me tons of sound advice. Jorge Just and Julia Longoria are editorial directors. And Bird Pinkerton watched as the boomerang whacked into the locking mechanism on the carriage. The door slid open. But the doctopus wasn't moving.

41:41Diana Deutsch:If you want to check out more about Diana Deutsch and Auditory Illusions, we've got a link in our show description where you can find more illusions to listen to and a ton of info about the illusions she's discovered. Thanks, as always, to Brian Resnick for co-creating the show, along with me and Bird. And if any of you out there have thoughts about the show, send us an email. We're at unexplainable at vox.com. You can also leave us a review or a rating wherever you listen, which really helps us out. And if you're into supporting the show and all of Vox in general, join our membership program.

42:10Diana Deutsch:You can go to vox.com slash members to sign up. Unexplainable is part of the Vox Media Podcast Network, and we'll be back with episode two of The Sound Barrier on Wednesday.

42:22Thank you.