Can we record our senses?

0:00This BBC podcast is supported by ads outside the UK.

0:30at Whole Foods Market. If modern life feels a little loud, you're not alone. At Miraval Resorts, mindfulness has never been about perfection or performance. For more than 30 years, it's been about presence and about slowing down enough to listen. At Miraval Berkshires in Lenox, Massachusetts, each season brings its own rhythm. Rolling green hills, quiet forests, and a sense of stillness that feels like a sanctuary. Here, well-being takes many forms, meditation, yoga, hiking, equine experiences, and time set aside simply to be present. These aren't just experiences for the moment. They're simple, intentional practices, tools you can carry with you long after you leave.

1:19Because wellness doesn't live in one place. It lives in how you show up again and again. Miraval Resorts, wellness for real life. Learn more at miravalresorts.com

1:36Hello and welcome to CrowdScience from the BBC World Service. I'm Anand Jagatia and you are just in time for Movie Night. We've got popcorn, 3D glasses and a huge TV. But that's not all, because this is a rather special home movie experience. On the screen in front of me, a battle is raging, complete with explosions and gunfire. Behind my head, I hear a little machine make a noise. Oh. And the room fills up with an unmistakable smell. Wow, OK. I can definitely smell smoke. But it's not because we've burnt the popcorn. is because we're watching a movie that has a smell track as well as a soundtrack.

2:24Lots of fire on screen and I mean I can smell burning so it's definitely working. I don't know if this is just because it's a hot day but I feel like the smell is also augmenting the feeling of heat from the fire. The smell is very powerful and really immersive in the movie. The inventor of this technology, Raul Porcar, is showing me how it works in his living room in Valencia, Spain. We're watching the sci-fi film Avatar from 2009. Right now, the bad guys in helicopters are destroying a lush rainforest. There's a huge tree about to come crashing down, sending branches, twigs and leaves flying everywhere.

3:09Mmm, OK. Okay, so now I'm smelling kind of fresh jungle kind of smell with rain. Really leaves? Yeah, like that kind of smell of freshly cut grass.

3:25I'm really surprised by how much it adds to the film. Like it does really make it feel more immersive. Once you have to smell it, it's almost like your brain thinks that it is more real. It's like you could actually be there. This week on CrowdScience, we're exploring the future of technology designed to stimulate all of our senses. Thanks to a question from one of our listeners in the USA. My name is Arvind Kulkarni. I'm based here in Fremont, California. I live with my family, my parents, wife, daughters and our pet rabbit, Nico. Oh, nice. That's a good detail. It was while he was out in nature one day that Arvind decided to get in touch with us.

4:09I walk quite frequently in my neighbourhood where there is like a park right in front of our house with lakes and everything and there is a creek that runs right next to our home. And as I'm walking through I enjoy the smells, the cherries, the oranges, the jasmines, the roses and whatnot. and one of the things that stuck me is we can easily record a scene with a picture or a video but we are not able to record smells and touches so I thought hey why not send this question to crowd science and see what you guys can dig up on this. We have video cameras and we have audio recorders but why don't we have smell recorders and like touch reproducers?

4:55Right. Are there any other smells or maybe tastes or I don't know tactile sensations that you would like to be able to record like if let's say we had magically one of these devices that could do that yeah definitely we travel a lot with our family to different national parks in the country recently we were in Alaska enjoyed the nature and the glaciers and everything there we have like tons and tons of pictures. I wish we could bring home along with the pictures and the videos, the smell and the touch aspects that would be wonderful to experience later on. What's so kind of like the sensation of walking over crunching snow or something like that?

5:36Yes, yes, yes. That would be amazing. When you touch the ice, it's literally like rock hard. It would be great if we can capture how the the cold wind sort of feels on your skin, how amazing it felt. Thanks, Arvind. I love this question. And it does make me wonder, more than 100 years on from the invention of the television, and even longer since radio, why do our media devices only encompass two of our senses? And how close are we to moving beyond our eyes and ears? Well, Arvind isn't the first person to wonder about the potential of our other faculties, especially when it comes to smell. Smell just impacts you so strongly, captures all of your attention, evokes things like memories and gut reactions.

6:26Jazz Brooks is an incoming assistant professor at MIT in the USA. He works in perceptual engineering, exploring how it's possible to manipulate our senses. And it's something that people have been thinking about for a very long time. Charlie Chaplin actually writes about scented films. It seemed to have been more common than expected. Not on the film production side, but on the film exhibition side. The film industry has been experimenting with adding scents to pictures for almost as long as movies themselves have been around. The earliest account is from this film impresario called Roxy Rutherford in 1910 in his small town.

7:07and he received a reel called the Rose Parade in Pasadena, California. And so he ran to the drugstore and got a rose perfume, dipped these like streamers into the rose perfume and placed it in front of the fans during the movie. And one female audience member comes up to him afterwards and is like, wow, it was so vivid, like the roses. It's almost like I could smell them. Yeah, I mean, maybe it works better if you don't tell people so they don't know. and yet they're like, oh, for some reason I just felt like that movie was really intense. And this wasn't something that was limited to him. Lots of other people did it, from things like sprays to odours in the air conditioning system, some of them with multiple odours.

7:50It really made it very vivid and intense, the experience. So these early innovations were pretty low-key. Sometimes they didn't even tell the audience they were doing them. But fast forward about half a century and smells in the cinema are about to take centre stage. You might even have heard of this one, smell-o-vision. For all the people who've heard the word smell-o-vision, it's become a joke, but it was an incredibly genuine effort to make a film where scent was a main part of the narrative. This is the scented storyteller, a.k.a. Tammy Burnstock. She does smell-based performance work in her home of Sydney in Australia.

8:28I do a lot of work with children that involve scent and story. And she and Jazz work together on a documentary about the history of scented cinema called Inglourious Smell-O-Vision. Jazz is in France and Tammy is in Australia, but we managed to coordinate our time zones so we could have a chat together. Thanks for giving up your time to your, what is it? It's evening I guess where you are, right? Or morning, I'm not sure. It's too early for me. Anyway, so Tammy, could you tell me the story of how Smell-O-Vision began? So happily. I mean, it's kind of become my life's mission. It was a very sophisticated system which no one's heard about, so I love the opportunity to talk about it on this show.

9:05Thank you. The story starts around 65 years ago, when two scented films were released at the same time and competed with each other at the box office. OK, so the Battle of the Smellies, as it was called in 1959 and 60, was this strange confluence of two scented film systems, Aromarama and Smell-O-Vision. So Aromarama premiered in 1959 in December, slightly pipping Smell-O-Vision at the post. And it was a documentary shot in China and it presented the scents through the air conditioning system. Then Smell-O-Vision happened in January of 1960 with a film called Scent of Mystery. The spine of the story is a mystery where the mystery woman doesn't appear till the very end except via her scent.

9:54The script was written especially for scent, and along with the soundtrack, there was a smell track. The technology that delivered those scents was a complex and expensive system that had to be set up specially in each theatre. A smell brain, invented by an osmologist called Hans Laub, pumped 30 scents to individual seats in the cinema. So people got whiffs of 30 different things. So there's this literal machine, you know, I'm imagining a proper kind of crazy scientist type contraption with dials and pipes. And you're saying that they released the smells during certain scenes to individual seats.

10:29How did they capture these smells? A lot of the smell experiences were basically hiring perfumers to do interpretations of the fragrances themselves. So it was like very much handcrafted work in this case. Yeah, I mean, there was a whole range of scents. There were very single note scents like peach and rose. and there was a scent joke around the taxi driver was having a coffee very strong and what the audience smelt was a Spanish coffee liqueur and there was banana. And so that was sort of the more simple, what you see is what you get. And then the more interesting stuff that was around character and emotion.

11:01And, you know, we've often compared the use of scent in films to soundtrack. So sometimes it's subliminal. Sometimes you don't even notice what the scent is, but it creates a feeling or a mood because I think that's the strength of scent, isn't it? that it can make you feel without necessary thinking. So you said it was a bit of a flop. Tell me about how it was received. So it was commercially not successful. It got a bad press at the time and was then made into a bit of a laughingstock. But we've spoken to quite a few people who actually experienced the film themselves and many of them say that it was actually an incredible experience for them.

11:37There were criticisms about the number of cents. So the system was not perfect. But having said that, I think quite a lot of people who went loved it. Back in 2015, Tammy and Jazz painstakingly recreated the Smell-O-Vision experience. And along with a team of perfumiers and engineers, they put on three screenings of the film in the UK, Denmark and, of course, Hollywood. We couldn't be pure and do the 30 cents that were in the original film. We ended up with, I think, 16 cents in all, a mixture of things that were atmospheric. and then kind of opt-in experiences where you can lean into the smell.

12:15Some of them were worn by two actors dressed up who sort of paraded in front of the screen and through the audience. So it's complex, but it demonstrated that there's definitely hunger for that kind of experience. What do you think the state of scented cinema is now? Do you think that it is just sort of this whimsical gimmick from the past or do you think maybe it has a future and that maybe in the future, yeah, when you go to the cinema, watch a film, it will also smell. I mean, I think it definitely has a future. There are different companies that are starting to do automated systems. I remember going to see Minions 3, I think.

12:53I went to go see it on 4DX. And I distinctively remember that some of the sequences had smelled. So it seems like it's on its way. It's happening. There's a hunger and an interest in doing these enhanced events that give people layers. It's not for every film, but certainly some films. It's an amazing experience, I think. So despite being around almost since the birth of cinema, Smell-O-Vision hasn't really gone mainstream. But could new forms of technology be about to change all that? That's what we'll be finding out next. This is summer at its peak. Whole Foods Market Summer Fruit Fest is your invitation to eat the season.

13:32Fresh, organic and bursting with flavour. Start your day with peaches and organic blueberries and yoghurt. Build a grazing board with fresh fruit, prosciutto, and artisanal cheese. Then fire up the grill with no antibiotics ever proteins and fresh produce. Savor the season. Shop Summer Fruit Fest at Whole Foods Market. If modern life feels a little loud, you're not alone. At Miraval Resorts, mindfulness has never been about perfection or performance. For more than 30 years, it's been about presence and about slowing down enough to listen. At Miraval Berkshires in Lenox, Massachusetts, each season brings its own rhythm, rolling green hills, quiet forests, and a sense of stillness that feels like a sanctuary.

14:19Here, well-being takes many forms, meditation, yoga, hiking, equine experiences, and time set aside simply to be present. These aren't just experiences for the moment. They're simple, intentional practices, tools you can carry with you long after you leave. Because wellness doesn't live in one place. It lives in how you show up again and again. Miraval Resorts. Wellness for real life. Learn more at miravalresorts.com

14:56I'm Anand Jagatia and you're listening to CrowdScience from the BBC World Service where this week we're trying to make sense of our senses. Right now I'm recording into a microphone so that you can listen back to my voice later. But CrowdScience listener Arvind in the United States wants to know whether we can do the same thing for recording smells and tactile experiences. Well, back in Valencia in Spain, Raul Porcar is trying to inject a breath of fresh air into the home cinema experience. He's the creator of a modern smell-o-vision system called Olorama, which you heard me experiencing at the start of the show.

15:37So, Raul, we just watched a bit of Avatar, quite an intense action scene, and now we're going to watch something a bit more light-hearted that's around food? Yeah. I don't want to advance anything. I want you to enjoy it. Raul has composed smell tracks for several different films, and right now he's about to show me the 2007 Disney Pixar film Ratatouille. So I have seen Ratatouille before a long time ago and from memory it's a film about a rat who wants to become a big time chef in Paris. It's a pretty seamless experience. Behind me is a black unit with a fan on the front loaded with smells that connects to the TV via Bluetooth so that at key moments during the film it can emit a little puff of intensely fragranced air.

16:24OK, so we've got some rats in a big garbage heap. Mmm, cake. Yeah, OK. That's a very distinctive smell. It works perfectly with a movie like Ratatouille, which is all about smell. That was the unmistakable aroma of cheese. And strawberry. And that is exactly what the rat is eating. Cheesy strawberry. That was really fun. So where did you get the idea to combine cinema or movies with smells? The idea of creating anorama brings to mind a Sunday watching TV with my family. They have a specific section in this show where they make some experiments. and these days something exploded and the presenters say, wow, I would love that you could smell like we do.

17:23So that turns on something in my mind and I started some research and after a couple of years I had my first prototype until today where we have units in the five continents. I should say we're sitting in your living room on a sofa in front of a TV and behind us is what, it looks like a kind of very high-tech fan with buttons on it. Is that what it is? Yeah. You have some cartridges inside and the fan helps to move the smell faster to get a good synchronization between the scent and the scene. And these are the same capsules. Wow, this really is like Willy Wonka stuff, okay? I mean, this kind of looks like a black CPU tower with a fan and then it's got these little cartridges that you can, so you just popped one out and that's the smell.

18:08What is that canister that you just popped out? That's apples. Apples, okay. and in terms of the smells like how do you get them how do you pick them well we have different providers they are specialized in aroma but some other specific like gunpowder or burning rubber we manufacture in our lab that's cool i mean i can imagine that like people who make perfume probably don't get many requests for the smell of burning rubber or like Yeah, that's the reason why we had to develop it ourselves. You might not want to wear a perfume that makes you smell of gunpowder or burning rubber, but they aren't totally unpleasant odours.

18:50Lots of people like the smell of petrol or the smoky tang of fireworks, me included. But Raoul says that there are some smells they just don't use, no matter how appropriate to the plot they might be. We start by trying with unpleasant smells. And the experience was very bad. The users really complain about this. The bad smells ruin the experience because the bad smells are more persistent than pleasant smells. Really? Yeah, our brain is very sensitive to this kind of smells. Today we don't go beyond this kind of smells. This isn't just a perceptual thing. it's actually hardwired into the brain.

19:33Unpleasant smells are processed faster than pleasant ones and with pleasant or neutral smells in the air we stop being able to smell them pretty quickly. It's called olfactory fatigue but that doesn't happen with nasty smells. We go on smelling them for much longer. But I imagine like a horror film would be great for this like you know the smell of like blood and guts. I mean it might be a bit gross but also if you're watching a film you kind of want to be grossed out right as you're insisting i will show your unpleasant smell okay okay go on then oh god what have i let myself in for so ral's just giving me a bottle of something which he says is unpleasant oh okay that is really bad actually i don't know what is it like is it is it is it sweat feet sweat yeah i was gonna say it was very kind of pungent acrid sort of sweat smell oh yeah it's still in the air do you think that smell is a really important sense but we often don't maybe appreciate it that much like i think culturally we're kind of much more fixated on vision and sound i wonder whether maybe you think more generally smell and aroma is sort of having a bit of a resurgence like the importance of smell it is actually a lot of research and investment in the smell but it's focused on fragrances.

20:55The perfumery is investing a lot of millions but in audiovisuals yes it's a forgotten sense but yes with the right investment we could find very good solutions and make impossible to have this device in your house and And you enjoy movies or videos with smells, which makes it more immersive. So, Arvind, to answer your question, there are people out there trying to make our media experience more multisensory and to augment sound and vision with smell. But even though the technology behind Raoul's system is impressive, smells in the real world are incredibly complex, made of hundreds of different odour molecules.

21:41So trying to capture and recreate them is challenging. Arvind also wanted to know about our sense of touch. Is that any simpler? Even though you might think of touch as one sense, it's actually lots of different sensations that overlap and intertwine. There are many different what are called mechanoreceptors, little cells that detect different kinds of stimulation. You can feel where things are contacting you, How hard are they pushing? Are they vibrating? Are they warming your skin or cooling it? Are they hurting you? All that information I would call the sense of touch. Our next guest is someone who knows all about the complexity of this touchy subject.

22:26I'm Catherine Kuchenbecker. I'm a director at the Max Planck Institute for Intelligent Systems in Stuttgart, Germany. And I lead a department called haptic intelligence. Catherine works in haptics, which is the science of the sense of touch. One application is known as haptic feedback, getting our gadgets to respond to us with a touch or a stroke or a poke. Imagine you're in virtual reality, you're wearing a head-mounted display, you can look around and see objects. But if you reach out with your virtual hand, in current systems, either you feel nothing or you might just feel a little buzz-buzz, similar to your cell phone.

23:01And so there are a lot of researchers that are working on developing haptic feedback technology to bring the sense of touch into these digital interactions like in VR or AR or on screens. Making VR gaming more immersive does sound exciting, but there's much more to this technology than that. Catherine is using it to help save lives in situations where humans use remote control robots to do jobs which are too delicate or too dangerous to do ourselves. a field known as teleoperations. They make these long, thin instruments that a surgeon just across the room can control very precisely to operate.

23:38For example, to remove a prostate in a very minimally invasive but also intuitive way. But these robots only recently do they have haptic feedback. When a surgeon is operating, say, on my stomach, I would really like them to be able to feel what they're touching. And so for a very long time, I've been working on bringing the sense of touch into teleoperation. motivated by surgery, but also other applications such as disaster relief. If we need to send a robot in to explore a building that's falling down, so rescue robotics. And so have you worked on the haptic equivalent of a photo? Like, is it possible to sort of take a kind of snapshot of how something feels and then simulate that again later?

24:21This is such a wonderful question. And it's actually something that has been motivating and inspiring me since I was a PhD student, a concept I called haptography, haptic photography. We create a tool with sensors inside and use it to explore textures like materials and make a digital model. And indeed, we were able to make synthetic canvas based on data recorded from touching pieces of canvas or other fabrics. We produced a database of 100 different material samples. Catherine's synthetic fabric library works using a specially designed little haptic stylus, a bit like a pen. You run the stylus over a smooth flat computer screen, but it gives you the sensation that you're running it over rough canvas or smooth silk or soft cotton.

25:09We can actually do a really good job of capturing what you feel through a tool because it has to do with the vibrations, like how the tool shakes back and forth as you drag it or tap it into different materials. But if you then instead think I want to do haptography with my fingers, that's current things we're working on now. That's still really difficult. And so how would that work? Would you have like a glove or like thimbles on your fingers that you would then like touch things with? We're actually exploring several different options. I don't ever want to interfere with the person who's doing the data recording because their skin needs to be present in the interaction.

25:42If we put a glove or a thimble on your skin, it's a different physical interaction taking place. So my preference is to add some little lightweight sensors onto your fingers. Like we can use cameras from all different directions and put little shiny balls, adhere them to your fingers so we can track the movement of your fingers. Even more difficult though is how to play those sensations for you to feel. I mean, all of that stuff sounds really exciting and it sounds quite futuristic. But on the other hand, it sounds like it's very much in its infancy. So compared to something like the way that we transmit visual and auditory information and, you know, the fidelity that we can get on a very high quality screen, touch seems like it's lagging quite a lot behind that.

26:24Indeed, I agree that we are far behind. I think it's because the problem is simply much harder. For visual feedback, your eyes are the only sensor, but touch, because it's spatially distributed throughout your body and it's multimodal and it's so tightly intertwined with movement, you need to be able to move however you want. So that's a much more complicated model than simply replaying a pre-existing experience. I'm going to bring us back to Arvin's original question, which is how far away are we from, for example, recreating the feeling of walking on the crunchy ice of a glacier and feeling the cold wind on your face?

27:05I think we have, for the most part, the sensors that can record, like who can put sensors in your shoes and measure what it feels like for you to walk across a glacier. Recreating that, maybe imagine like a virtual reality cave, you need ways to make it warm and cool, you had to do it synchronized with the wind and some actuators on your shoes. I don't know how compelling it would be, but I think we could put something together. Now, if you then want to reach out and touch the ice, I think that's much more difficult because we have to be ready to respond to however you move. What if you go touch the ice with your elbow or with your nose?