Bonus Episode: Inside Sonos's distributed research & design process for the Arc Ultra

0:00Matt Benatan:On the personal front, quite often we'd be watching content with family members who might be struggling with dialogue. And so looking for ways to enable that for people so that they can enjoy content with loved ones is something that we're very happy to have been able to do. And it's one of the key motivating factors behind the RNID collaboration.

0:22Naphur van Apeldoorn:For many creative people, music is a central part of life. You can count me in that camp. I have long admired Sonos for their quality of their products and easy integration of their products into my home, and everything they design seems to be beautiful. The integration of design and technology is often a challenge for companies because they've got different groups with different values trying to work together. But from the outside, it would seem that Sonos has something figured out in this department. To learn more about their approach, we spoke with Matt Benetton, Principal Research Scientist and Nafur Van Appeldorn, Senior Manager of Hardware Development and Engineering.

1:05Naphur van Apeldoorn:We chat with Nafur and Matt about how Sonos decided to invest the time and energy into a new product, Arc Ultra, and what the R &D and prototyping process looked like. As a distributed team, they share physical prototypes using a 3D printer. We also talked about how they're using onboard AI for speech enhancement and why it seems like movie dialogue has become harder to hear over recent years, even if you don't have kids making noise in the background while you're trying to watch. This is Design Better, where we explore creativity at the intersection of design and technology. I'm Eli Woolery. And I'm Aaron Walter.

1:38Naphur van Apeldoorn:At Design Better, our primary mission is to produce work that helps people like you refine your craft, improve your collaboration skills, and get inspired by the creative process of others. If you enjoy what we do here, the best way to support us is to become a premium subscriber at designbetterpodcast.com slash subscribe.

2:08Naphur van Apeldoorn:We'll return to the conversation after this quick break.

2:18Naphur van Apeldoorn:And now, back to the show.

2:25Naphur van Apeldoorn:Nafur Van Appeldorn and Matt Benetton, welcome to the Design Better podcast. Thanks for having us. We're excited to talk to you. Aaron and I are both really big Sonos fans. I've got quite a few of the speakers, and Aaron has even more. So many, in fact, that he doesn't know quite exactly how many he has at his house, which is pretty cool. Let's start with this new product that you guys released. And we want to figure out a few things. I mean, for our audience that's sort of within the design and product design realm, we thought it might be interesting to start from a user research perspective.

2:58Naphur van Apeldoorn:and maybe you could talk to us a little bit about how you did the user research. Also, I believe one of you was prior on the RRD team too, so maybe that was part of it. But how did you decide to invest that time and energy needed to come up with this new product?

3:12Matt Benatan:Yeah, so I was part of the team which developed the sound motion transducer in Arc Ultra. And it's really cool how that happened. So I'm now part of Sonos for three years. And that's also when the acquisition happened of the Mite technology. and basically three days after the acquisition we were flown to Santa Barbara one of the headquarters of Sonos is over there and we joined the room 30 engineers user research and all kind of other teams were there and they had like this arc the previous soundbar on the table and they had our transducer technology right next to it and we're like okay how can we fit like this technology inside the soundbar and we were like okay why do we want to do this and like all this kind of questions, you know, but there was this ambition of creating a new sound experience, which was previously impossible without this technology.

4:01Matt Benatan:And then like, what is this technology about or the sound motion? That's really about creating more bass from a smaller package. That's essentially what it enables you from a technical perspective. But what it does from a product perspective that becomes like very interesting is because you have now a kind of dedicated subwoofer or lower frequency device in your soundbar. The other speakers can focus on the vocal range, the placement of audio within the room, and that allowed us to make it like a truly Dolby Atmos soundbar with a 9.1.4 representation inside the room, and that allows you to deliver Dolby Atmos content inside everybody's living room.

4:38Matt Benatan:That's the true innovation here, that we could enable that experience for our home users.

4:43Naphur van Apeldoorn:So for folks that might not be familiar with Dolby Atmos, is that what you'd hear if you go to a cinema, like typically go to the movies, that's the type of technology they're using there.

4:52Matt Benatan:Exactly. This full sound bubble, not only from the front, left, right, but also the upper channels. That's this whole Dolby Atkins bubble we can deliver. It's really in the nature of Sonos of delivering that kind of content to users. If you already go back to where Sonos was founded 20 years ago, you should realize back then there was no Spotify, Apple Music, you know, there's music streaming wasn't at the level where it is today. And they decided like, hey, we want to deliver music content to people, their home, everywhere, anywhere, all the time. It started with like the all-in-one products.

5:27Matt Benatan:But nowadays we have soundbars, wearables, portable devices. We have a whole wide range of products where we can deliver music anywhere to people. And we believe like this Dolby Atmos content is the next content we want to deliver to people. Like you have now Spotify, you have Apple Music delivering Dolby Atmos content, you know, and people need devices to play that content as well. It's not only music, but it's also movies, theater. And this is our promise on delivering that experience to people their homes.

5:53Naphur van Apeldoorn:Nafur, you alluded to an acquisition that I'm not familiar with. What was the acquisition that Sonos made that brought this technology into the company?

6:02Matt Benatan:So that was the acquisition of Might, which was a startup based in the Netherlands and was founded by two brothers, Matthias and Timothy Schick, they invented technology where we can make low frequency speakers five to six times smaller than conventional technology. And that's like the technology which Son's acquired three years ago. And it's amazing like how that's now available to everybody inside the ARK Ultra product. And to give a reference of what kind of leap in technology we made over there, the previous ARK is and was an amazing product, but that one had like 11 transducers inside that sound power.

6:36Matt Benatan:and with Arc Ultra we were able to fit like 14 transducers in that single soundbar. And then if you dive a bit deeper what those transducers consist of, inside the previous soundbar we had eight woofers and three tweeters. And those eight woofers were responsible for the mid and low frequency and those tweeters for the high frequencies. And because you only had three tweeters and those woofers were also responsible for the bass, it's very hard to position sound as the Dolby Atmos bubble in the room but also have that vocal clarity, which the AI speech extraction is really a good wit as well. With the Arc Ultra, we have 14 transducers.

7:12Matt Benatan:One is the sound motion transducer based on the MITE technology, which Sony has acquired. We have six woofers, which deliver that mid-range frequency. And we have seven tweeters, which point the sound everywhere in your room. And that gives you that Dolby Atmos content.

7:26Naphur van Apeldoorn:Yeah. Matt, so your background is in machine learning and AI. And this is fascinating, the way that y 'all are bringing that into audio design too, because there's a lot of different reworking and reinterpretation that can happen in the devices to deliver a different type of sound experience. Can you tell us a little bit about how the software inside of devices is playing a role here?

7:53Matt Benatan:One of the things that came along with Arc Ultra was a bit more processing capability. So before we get to the software, we need to make sure that we have the hardware that we actually need to support the stuff that we want to do. And with Arc Ultra, we had more horsepower. And during the development of Arc Ultra, there was some work going on within what was then the advanced technologies team at Sonos, where we were looking into techniques for speech extraction. So speech extraction is part of the broader field, if you like, of source separation. And these are the kinds of algorithms that you'll encounter them kind of day to day, particularly if you're on Zoom calls, even on telephone calls.

8:35Matt Benatan:These are the sorts of algorithms that remove that stationary noise and provide clear speech signals. And for years and years, these have been developed in telecommunications. But applying them within film and television content is much more challenging because in this domain, we've got just a much, much broader variety of different types of sounds. And we're also kind of fighting with the fact that these are intentionally crafted. If you're struggling with dialogue and you're watching a film by a well-known director, it's going to be designed to be that way, right?

9:08Naphur van Apeldoorn:David Lynch, for example. Impossible to hear.

9:12Matt Benatan:Exactly. And there are many, many examples of this which I and others have struggled with. And that's before we even get to the point of hearing loss. The technology that we were investigating in the Advanced Technologies Group was a kind of advanced version of the sorts of source separation that had been used previously. And they were more sophisticated. They were using deep neural networks, so using machine learning, using AI, in order to be able to adapt frame by frame to the audio and understand where the speech content was to be able to draw that out. We realized early on in this process, so just to acknowledge the initial work was carried out by my colleagues, Hadid Murabi and Chris Pike.

9:51Matt Benatan:and when looking at these early prototypes we realized how much potential there was to change the way that we deliver speech enhancement on Sonos products and specifically on the ARK Ultra which a big part of my work was making sure that we were able to deliver that that we could do things like adapt to all of the different kinds of scenes that you would encounter within film television audio and be able to do this with something that was able to run on device in real time in order to deliver an excellent level of speech enhancement but enable a better sound experience overall and something that people can and attune to their own needs right so we've got these four different levels which is again something new so with Arc Ultra we introduced three new levels of enhancement and with this latest update you've now got four levels this This is made possible because, again, coming back to what we're able to do with the software, we're now able to not just use things like equalization or compression techniques that are commonly used in speech enhancement.

10:57Matt Benatan:We're able to actually pull that speech out and use that capability as a tool within the speech enhancement feature in order to understand where the speech is and use that to really emphasize the speech content, you know, according to the particular level that the user chooses.

11:13Naphur van Apeldoorn:You mentioned the difficulty in hearing dialogue in modern movies. That certainly resonates with me. I mean, in our family, it's also because our kids are loud, but we often have the captions turned on. And I'm curious if either of you have more insights on why that's become the case. I mean, I've heard things like, well, directorial styles like Christopher Nolan, or back in the and 50s, you could count on an actor like a Cary Grant having this sort of made up mid-Atlantic style of speech where they were very crisply enunciating everything, or maybe they had a theatrical background and so they were projecting in that way.

11:46Naphur van Apeldoorn:But now maybe the acting styles have changed. Are there other technical things that either of you are aware of for the reasons that we have trouble understanding hearing the dialogue these days?

11:55Matt Benatan:There are all sorts of things that contribute to this. You're absolutely right. There are things about the way that content is created now and the way that actors are directed you know lynch and nolan are people who want things to be gritty they want things to be real and when you're creating those environments one of the things that you might want to do is not to have the dialogue jumping out right you can kind of understand these sorts of decisions but on top of that with just the volume of content that's being produced today there are some smaller studios and stuff where you know maybe they don't have as much time to work on dialogue and as such you know there are things that kind of get rushed through particularly when you're talking about not obviously your big films and stuff like that but your more casual tv programs in terms of why people want to use dialogue enhancement there are all sorts of environmental reasons that dialogue might be more difficult to hear right you might have the air con on you might have someone doing some noisy diy next door well as you said right your kids could be loud.

12:55Matt Benatan:All sorts of reasons to use it that aren't just based around the content.

12:58Naphur van Apeldoorn:Talk to us a little bit about the hearing loss aspect, because you had an interesting partnership around hearing loss and accessibility that I found fascinating, especially for a company like Sonos. It was a little unexpected.

13:09Matt Benatan:Let me start at the beginning of the story, which was, as I said, we had these prototypes of what this speech extraction could do. And at that point, myself and my manager, James Nesfield, we were talking through what this could mean for the future of a speech enhancement feature. And there were people in our personal lives who we felt could really benefit from something that went beyond what we had experienced was commercially available at that time. James's mother and my father-in-law both struggled with dialogue. My father-in-law had been using hearing aids for around a year, I think, at that point.

13:44Matt Benatan:And if you know anybody who uses hearing aids, you kind of need to spend a lot of time working with an audiologist to get them tuned properly so that you get the best experience. So when you start using those, you don't really want to have them in when you're watching content because they're probably not going to be giving you the best kind of sound experience.

14:01Naphur van Apeldoorn:And you can get overwhelmed because, you know, if it's an action film, it's like quiet dialogue and all of a sudden it's explosive, overwhelming sound.

14:10Matt Benatan:Absolutely. And that touches on another really, really important point. So the charity that we worked with was the Royal National Institute for Deaf People here in the UK. They're a leading hearing loss charity. One of the things that we learned from them touches on exactly that point that you mentioned about dynamic range, which is a phenomenon which is called loudness recruit. So loudness recruitment is this kind of awful phenomenon in which not only do quieter sounds become more difficult to hear, but your threshold for louder sounds reduces, So it's compressing that dynamic range of comfortable sound.

14:46Matt Benatan:And as a result of this, we learned that, okay, having more control over the speech will allow us to do more to kind of compress that range of the overall audio and deliver an experience that allows people to still be immersed, but to be able to hear that dialogue and to be able to engage in it. Learning this stuff from them was really important, right? The RNID taught us a huge amount, but so much of the value came from the participants that the RNID recruited to take part in the design process. So we had several phases of listening tests. We would invite people to come in and trial prototypes as we were developing this in order to understand what people really wanted, what people needed in order to be able to enjoy content.

15:30Matt Benatan:And ultimately to get their feedback on the feature as it ended up.

15:37Naphur van Apeldoorn:We'll return to the conversation after this quick break.

15:44Naphur van Apeldoorn:Design Better is supported by Masterclass. For a while, I had this nagging feeling that I was busy all day but not actually making anything. But then I took Cal Newport's class on Masterclass about rebuilding your focus, and one idea stopped me cold. Your attention isn't being lost, it's being taken. And if you don't design your day, someone else's app will design it for you. Now I schedule my creative work like a meeting that I can't cancel, and the difference in my output has been remarkable. That's the thing about Masterclass. It puts you in the room with the people who defined their fields, not just experts, but the best of the best in the world.

16:23Naphur van Apeldoorn:There are over 200 classes across business, writing, creativity, wellness, and so much more, with plans starting at just$10 a month, billed annually. Right now, as a listener of Design Better, you get at least 15 % off any annual membership at masterclass.com slash design better. That's 15 % off masterclass.com slash design better.

16:49Naphur van Apeldoorn:And now, back to the show.