Teaching Machines to Smell with Osmo CTO Richard Whitcomb

0:00What's always been missing from the industry is how to store the smell in the middle. How do you store, like you store pictures and pixels, how do you store the representation of smell in a way to recreate it? And what's really hard about smell is unlike vision, which is three dimensional, three or four dimensional, but there's around 300 different receptors. So the amount of dimensions you need to capture smell is probably in the order of hundreds, which means you really need modern AI and the ability to do embeddings and data to represent it in that way. Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters.

0:39I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI.

0:49We've given AI sight, we've given it speech, but what happens when you give it smell? And that's the question at the heart of today's conversation, because for all the progress we've made teaching machines to see and speak and reason, they still have no chemical awareness of the world around them. And my guest today, Richard Whitcomb, is the CTO of Osmo, a company on a mission to change that. And they're building AI that can understand and even generate scent. It's wild. He's also a former engineer at Twitter, Spotify, and NVIDIA. So this guy has some good pedigree behind him as well. And this is a seriously fascinating frontier in multimodal AI where digital intelligence is meeting the physical world.

1:30In this episode, we're going to dig into how Osmo is teaching machines to smell and what it means for the future of robotics, health and everyday life. But welcome to Talking AI, Richard. It's great to be here today. Yeah, excited, really excited for this chat. But I got to start here. How did you get in the field of giving computers and machines a sense of smell? How did that even come about as a domain that you got into? Yeah, so it's a funny background. Alex, the CEO, he and I used to work together at Twitter on a research team there. He's the one he went to Google and that's where he started this foundational work on smell and how we digitize smell there with as part of Google brain at the time, which is now Google DeepMind.

2:11I went to Spotify, but after a few years at Spotify, I joined NVIDIA. And what I was working on NVIDIA was self-driving cars was the primary project we were building the platform for. So at NVIDIA, I spent a lot of time working on all sorts of different sensors like LIDARs, cameras, ultrasonic radar, all the multimodal sensors that you need to actually have a car navigate the world. But then I had dinner with Alex and he was talking about his research and it really got me thinking, why can't I put a sensor on the car that tells the car about the smells or really like the chemical environment? Why can't the car figure out if there's like a gas leak, if there's some type of warning?

2:50The world of chemistry around this tells us lots of things why humans have a nose is to tell us is there a good things bad things should we run should we move towards food so there's a lot of information in the world so through our conversation they got very curious about why can't i just buy a sensor like a lidar that i put on the car that tells us this so that got me started then he reached out again later and was like you know what i'm going to start this company spin out of google and really go after digitizing smell so that first conversation got me interested and that was the hook that got me in because I got more and more into reading and learning and figuring out what's the gap there and why can't we do this yet why can't I just purchase a sensor put it onto a car into my house into my phone possibly yeah that has this information to me because there's so much we can learn and computers and AI can do with if this was available to computers in a very holistic way and we're going to get into some of the use cases the use cases are wild so I guess self-driving cars Nvidia that wasn't ambitious enough.

3:49You had to go tackle the smell. But on that point, the multimodality has been such a huge component of a lot of the advancements in AI where they're using one modality or modalities in conjunction as inputs, outputs, all of these different things. But why have we struggled with smell? Like, why has this not happened yet? Because we have always, as a kid had the imagine the TV would have like smell-o-vision, right? And you get to smell what's on the TV, but why has this one been so difficult? there's a few reasons for this one historically if you think about audio or you also think about vision too between a video camera and also a monitor to display audio of course you have a microphone and also a speaker to do it too so those have been around a long time but also we've done the fundamental works a long time ago to figure out how to represent audio and also vision in computers yeah and how do we store it like rgb was actually a lot of early work so i'll go back hundred years we actually didn't know how to represent vision in a way rgb being just green red blue and decor kind of color yeah in the primary or yeah their space too so there was work that had to be done to actually just understand what it means to store images there's anything with audio we had to figure out how to store audio sounds too so there was work and then based on that work you can start to figure out how to record let's say sound or video and also how to replay it, but also how to store it in a way that you can redo it too.

5:17So there was a bunch of work that happened there. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and they're ranked by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free.

5:49If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI-opportunity-finder. With smell and just chemical sensing in general, there actually are sensors who do a really good job of capturing this. They're not quite yet like a camera. They tend to have to be in labs, but these sensors have been around since the 1970s, GCMS is the biggest sensor that is used in the industry to do this. A lot of forensics and different places actually have these sensors. And of course we can make smells, which is mixing different ingredients together and chemicals together to create smells.

6:28That's the whole fragrance industry is the creation of smells. What's always been missing from the industry is how to store the smell in the middle. How do you store, like you store pictures and pixels, how do you store the representation of smell in a way to recreate it? And what's really hard about smell is unlike vision, which is three dimensional, three or four dimensional, depend on how you think about it. How we smell is actually receptors in our nose. And there's actually varies a bit. We haven't fully nailed down, but there's around 300 different receptors that mapped it. So the amount of dimensions you need to capture smell is probably in the order of hundreds, which means you really need modern AI and the ability to do embeddings and data to represent it in that way.

7:15Unlike back in the 60s and 50s, you can actually figure out how to represent video images in a way. But smell, it did take modern AI and deep learning to really get to the point where you can embed and capture this in a meaningful way. Yeah, and so when you have 100, 300 different, And then the combinations are just like mind blowing in a sense. So that's where AI comes in. Has it been that this, obviously, generative AI coming onto the scene has unlocked so many things? Was there an unlock from just traditional AI machine learning, all of those things to what we know today is generative AI, a transformer technology that was unlocked in this field modality of smell?

8:02yeah the big thing that the gen ai part has unlocked for us is actually the generative part so where we use generative most is we are using more i would call traditional techniques like embeddings and deep learning to to represent the map of smells and represent this in a way to get those like you kind of think about it as the rgb the pixel values so we use deep learning there what we can do now though is on the fragrance side one of the core industries we're kind of opinion is the fragrance industry to create new fragrances. You can think about these as like perfumes, but also beyond that, like the smells that are in the candles you use every day, the shampoos, the hand soaps, they all have a fragrance and a company behind that creates that smell.

8:45So we are using Gen AI techniques to go and create new fragrances and smells too. So like our customers can come in and say, they describe the scent that they want. then we can use generic techniques with these embeddings and its map ins to create a fragrance that best maps to what they want at the end of the day so you're almost in that use case right there and i want to dig deeper deeper into this fragrance one so the input is effectively text essentially and the output is smell a fragrance that can be created right yep it's a recipe a recipe yeah yeah so like again like the speaker is divine so like the recipe can go to a big robot that mixes ingredients together that creates the fragrance that you then hand off to the client and they smell it.

9:31And then they can actually manufacture large amounts of it and they can go put it into their products, which is what the fragrance industry does. So do you see, there's so many crazy products coming out, like Suno for the music side of things where any user can go and generate their own music. Do you see a future where any individual could create their own custom fragrance. Is that like a whole additional market where you can create your own fragrance and then obviously, you know, the machines and robots go and build it and it gets delivered to you? Yeah, that's exactly what we're building right now.

10:02So we're building a new product called Osmo Studio. That's in a closed data right now that allows exactly for this. So you can go into Osmo Studio and describe the scent that you want. You can also upload images. You can describe like different notes or maybe different time day or different longevity. And then from there, we send you out a sample kit of three possible formulas or recipes that match your description then you can pick one of those then work with us and actually scale up and get a whole bunch of them like you can actually order 10 to 100 fragrances that you then can keep for yourself give to your friends or in the future sell on an etsy store or through website too so we really want to democratize the idea of i want to create a scent there's something i want in the world and And I want to either pray for myself, share with my friends, or even start a business on top of it too, and make that much more accessible.

10:53Everything I'm saying right here is possible today, but you really have to spend a lot of money to get personalized time with a perfumer who then will listen to you, walk through it. And so if you have$100 ,000, this is all possible. And you can get like 100 kilograms of oil created for you. A big barrel is how it works today. But we're trying to make it much more accessible. So you can actually create custom scents for different purposes. and get it ordered. Then also move beyond that. So like right now we're focused on fine fragrances and things like body sprays and hairsprays, but in the future going into things like candles and shampoos and laundries and kind of, wherever you want to create a scent, we can do it for you too.

11:33And that's one of the goals. The candle sign, my wife would sign up for that tomorrow. She started making candles at home, but having that element of, okay, here's my scent that I created too that I'm going to put into the candle is just wild. But here's a, and probably not off the wall for you, people have different tastes in terms of smell. Like this may smell good to me and it may smell bad to you, or I may like this scent. Is there like a chemical reason for that? Or is it truly just people have different tastes? Is there like something you can get to atomically digitally that kind of dictates that?

12:10Yeah, there's actually many layers to it. then to say look i'll try to stay a little high level but so one side of it one thing we that has been confirmed in kind of the olfaction neuroscience community is if you do train panels of people especially with very different backgrounds across the world if you train them to references so if you teach them that this smells like cherry yeah they will consistently tell you that this thing smells like cherry so you can actually get as close as possible to like round truth objective labels on top of data that being said though people like people are color blind there's also different changes in the receptors and then those where you can't smell certain things that the cilantro is a famous one for this whereas like it tastes soapy to people which is a smell yeah oh that's wild yeah so some personal preferences may come down to your receptors are slightly different than someone else's so you may like a smell where someone else does not like the smell.

13:04We actually don't know how much, but it's probably like single digits percentage of people have different variations of this. So from an AI standpoint, you can collect data, but if you have a large enough panel of people, it tends to wash out. What is definitely true though, is cultures have very different expectations of smell. A very concrete example is cinnamon in Middle East smells clean. In the US, it's citrus. yeah so if you put cinnamon in a soap in the u.s it would smell like christmas to the people in the u.s but it would smell clean in the middle east so what you like and what's pleasant actually varies immensely across cultures so there's the nature element as well that plays into it oh that's wild so the fragrance use case is like a really interesting one from like a commercial standpoint, like I instantly understand the value in the use case.

14:00Y 'all have done some wild things, one with mosquito repellent, which is not something that would initially trigger in my mind as like a use case. But give us that one, for example. Yeah. So one of the other things we do here is maybe one step higher is it's actually very interesting to think through what is smell. yeah yeah so what does it mean to smell things when you smell things it really is chemicals like very little molecules floating in the air that go into your nose that you smell a thing has to go into your nose bind to a receptor and that's what produces the smell at the end of the day so what it means if this shy smell is means that we work very carefully with what chemists call volatile molecules what that just means is these are molecules that can come off things and get into the air very easily.

14:52And they tend to be very small molecules too. So another line of business we have here, Osmo, that really is coming out of the fundamental research and the science paper the team did at Google Brain is we can also develop and discover new molecules. So we can predict what molecules will smell. So we can go and find molecules that don't exist yet in nature and bring those to market. Oh, wow. And we really focus on smell. So it's really what that means is molecules that get into the air. So on one side of the business, what we do here is we actually develop new molecules for the furnace industry.

15:26So new things, new smells we could put into candles and soaps. But using the same system, we can change one type of endpoint from what does it smell like to a human to does it smell bad to a mosquito? Yeah. Which makes it a mosquito repellent. Yeah. So like very different olfactory receptors. So insects and humans are very far apart. But if you have the right data, you can train a similar type of model, predict that. So you basically can use the same system that's trying to find good smells for humans, finds essentially bad smells for mosquitoes. Yeah, yeah. And then from there, you actually find new repellents.

16:04And then you can actually go to a chemical company and say, hey, we found a new repellent. We also have other models that make sure it's safe for the humans, for the environment. So we can go and say, Hey, we found two or three molecules, which are more powerful than D and also safe for humans and for the environment and move forward trying to commercialize that molecule too. That's such a cool use case. And it's such a good use case for AI where you're putting these inputs in and then you're getting something that's better on several fields. But it's almost like you're taking the, and I think you mentioned this earlier, it's going from atoms to bits and just thinking of this like chemical, invisible chemical environment all around us is just wild.

16:50That's nuts that you could actually create new compounds. I don't know if that's the way you describe it in a sense. So it's like you're almost inventing something new from a fragrance. What's the term olfactory? type of thing as well. Olfaction. Yeah. Yeah. Yeah. It's interesting when you think about it, a lot of things like, because it's funny with Morgan Osment, you see this because like smell feels so almost like magical when you say smell, it's just something that humans do too. But it's really exactly what you said. It's like what we really focus on is going from atoms to bits, which is the digitized smell part, then bits to atoms, which is the creation part.

17:29Right. So that's what a microphone in the speaker does, right? It goes from essentially waves to. Yeah, totally. Yeah. Digitize it then recreates the waves again. So we're doing the same thing, but in chemicals as opposed to sound waves. Yeah. Okay. So the next use case that I think is just wild is you can use scent to detect counterfeit items, which is just completely wild. And this is a huge, I would assume opportunity because counterfeits are widespread across all kinds of industries. He's like, how does that work where you can smell something and then say, oh yeah, that's fake. Yeah. So I think about it too.

18:06It's actually in those cases. So we have done this before. If you have counter items, let's call it a shoe or handbag or wherever it is. You human, like I can sell the difference. If you, if I handed you to two things, you can actually smell the difference. But you'd have to have both of them together for comparison and then know, okay, this is the real thing, right? Yeah. Yeah. And what it comes down to is what I said earlier is it's going from atoms to bits. There are the sensors in the lab that can do this, but we also are developing really on top of technology that's been around since the 1970s, like building AI models on top of existing sensors to do this.

18:40And so we're able to do it much faster in a factory setting too. And how this works is, kind of what I said before, when you have some material or whatever it is, smell is just very little volatile chemicals or compounds coming off of that and getting into the air. so if you have a fake factory that's making a fake handbag they likely don't have access to the same glues and materials that the real factory does too so when you put the fake a lot of it's actually the adhesives and glues so when you put the adhesives in the fake product the compounds of chemicals that make up that adhesive release different follow chemicals into the world and then the sensor can detect that and tell the difference.

19:24So the AI trained on top of that sensor can learn what is fake looking and what's real looking, and that's how you get the detector. That's why. And I got to say the fake merchandise is getting so good from like the box to the certificate, because you used to say, oh, I have this certificate of authenticity, but that fake is getting so good. And I'm assuming the large brands like the handbags and things like that's eating into their business when the fake is just as good as the real thing. But I could see this being the, I almost see there's some type of device or something, or maybe it's in your phone where you can say, okay, push a button and then, oh, that's fake or, oh, that's real.

20:04And then that is insane. I could see that being insanely valuable to those businesses. But like to that point, that doesn't exist today. Where do you see this going? I almost feel like there's platform play in a sense or something along those lines to where this can be leveraged in a composable, multimodal way in a sense, right? Yeah, that's one of the long-term visions of Osmo. Right now, we kind of have to do everything vertically because the sensors really aren't there yet. So we kind of have to develop. And likely to get better and better at this, we'll probably have to mix different sensor technologies together.

20:43So you have to go vertical. And at the same time too, when you go on the other side and you try to recreate the smell, that also exists, but we have to build a lot of customers tools that have vertically. But long-term for me, going back to the example I said when I was working on Soul Charming Cars, I really want, and I think the long-term vision of Osmo is this should all be possible as a platform. So like we should actually, when we digitize smell, that means anyone should be able to purchase a sensor and then connect it to any other AI agent model and add another multimodal thing, which is smell.

21:17Yeah. So if you're building a robot that's working through a house or in a factory, it should be able to know if there's a chemical spill or something else that's going on around it. We're focused right now fully vertically because we have to be to get this all working. But long-term, the hope is it just becomes another thing like hearing a vision and it just works. You could add it to places where it makes sense to add it to. Yeah. It's funny. I'm doing a presentation at a conference just on the topic of AI and I always said multimodality. And I tell people to think about it like senses. And I have the categories of vision, speech, all of these different ones, but I always say, Hey, we haven't gotten smell yet.

21:55So I need to add the new box with Osmo in there. But the other one obviously is the humanoid robotics side. You have like figure Neo just came out the other day where you can actually, anybody can purchase these things and bring them into your home. So that's the like obvious use case there is that sensor gets embedded into these robots. And then the use cases are wild from, Hey, the house smells bad. Let me go spray some Febreze to, oh, there's a gas leak that humans can't detect, but I can now you need to get out of the house. Right? Yeah. Yeah. There's so many things. There is a really strong reason evolutionary why like almost all animals on earth can smell in some sense.

22:39Right. because like otherwise it wouldn't exist. So computers not being able to do this actually does limit some cases. My main favorite example for me for houses is always, I would love to have a sensor in my house that just tells me like, do I have mold in my house? And if I do have mold, what type of mold is it? And hopefully where is it also? Like everyone knows, like when you walk into a house, you can smell it. Maybe if there's mold or something else in the house too, but. And how dangerous is it? I'm sure that could be digitized in terms of that being a factor as well. Yeah, yeah, exactly.

23:12But there's so many use cases that we also talked about use cases, like even a sensor that helps you, are you cooking the food well? Is it overcooked? There's so many things that like humans just do like neatly where you smell things going wrong in the kitchen. Well, back to food, like food spoilers, right? We always rely on this date, but maybe it actually isn't spoiled. Maybe it's still good in a sense. Yeah, exactly. And the long-term vision of a company too is the sensor that we have right now that we do the calf detection on, we roughly think about equivalent to a human nose too. But the goal also is to go beyond that and actually get closer to a dog's nose.

23:45Yeah. When you think about what a dog could smell too, then you get into things like disease detection and kind of finding humans away find it. And so there's whole sorts of category of things that like, now, once you think about a dog can do things that are looking in a sense like magical compared to what a human can do. I think smells like the sense that we take for granted as humans, probably because it is invisible. But to your point, like other animals, like dogs and things like that, it's critical to their wellbeing, but I didn't realize they could, what do you mean by smell disease? I'm sure it goes back to the same logical point you pointed out before, but what's the indicators there and like what type of diseases can you potentially identify?

24:24So roughly there's many cases of this. There are actually some humans who can smell Parkinson's disease, and that's been actually very well studied too. Wow. But there's also dogs that can smell cancer also. And what this really comes down to is changes that happen inside your body are often reflected. They come out. Like when you have cancer or disease states, different part of metabolic pathways and things change in your body. Maybe it's just a very slight way. But that's enough usually to actually change the kind of chemical signature that's coming off your body. Yeah. So there's many cases of dogs being able to train to detect this too.

24:59But again, the dogs aren't doing anything in a sense like supernatural. There are changes in some chemical that's coming off of the human body that the dog can recognize that there's no real reason at its limit that a sensor couldn't do the same thing. Yeah. It's essentially, we don't know what we don't know. Like, I don't know that smell indicates cancer or Parkinson's or whatever it is, essentially. And this is enabling that in a sense. But that's, that is wild. So now you've got the humanoid robot that can spray Febreze, identify mold and say, Hey, by the way, I think you may have cancer. That's insane.

25:38Possibly some point in the future. And then early diagnosis is huge in a sense before things get too bad. But I'm curious to your take here. It's a little bit tensioned until, but the head of back to Google at DeepMind, they're talking about, Hey, we may be able to cure all diseases through AI. Curious your thoughts on that. Do you agree with that? What does this future start to look like potentially? So the real hard part, and a lot of companies starting to focus on this too, is AI. It's always been this case too, software can move very fast. Maybe a concrete example for Osmos like this is, so we are able to screen billions of compounds in a computer in a few seconds to look for different repellents and kind of like maybe new smells like maybe a new woody smell that we want to find but really at the end of the day that gives us like a list of things to look at more carefully yeah the real expensive slow part is okay now we actually have to make this like digital version of it a reality so yeah in our case we actually have to have a chemistry team either internally externally go and synthesize that molecule so we can finally smell it and test it so robotics is like really focusing there too moving from like the bits to the atoms the atoms move so much slower than the bits so when you think about curing diseases and really allowing ai to try experiments and try things how it interacts with the world is very important because many things you kind of have to create in the world and test and see how it does in humans and things so maybe eventually as we get better at this we'll have full simulation in software and AI.

27:20So like, you no longer have to do synthesized things in terms of chemistry because it's just known. But I think we're very far from that. Like chemical prediction, if a synthesis path will work, can actually make this thing that I found is actually very difficult still at this point. And it gets much, much more complex when you get into like clinical testing. So will this be safe and effective in humans? There's still, there's still, the human body's complex, right? So there's lots of steps there too. A lot of the focus on it will be, I believe now or even very soon, AI would be very good at presenting possible drugs or therapeutics or like small molecules to try for help.

28:02But how you actually test it and get it through the FDA and approval is a big unknown. How do you do the Adam part of it? Yeah. It's a challenge. It's interesting. You mentioned the synthetic data element, like back to the Nvidia stuff. I've heard of them kind of having these synthetic, just not real environments where they're helping train robotics in a sense, living in the world. And you had the same thing with Google kind of creating these real world simulations, which are neat when you think about the future of gaming, but it's also additional training data in a sense. But I'm curious, once you've built at Osmo, is there an actual model, like we think traditionally, like a large language model that you are developing or where does that come into play or is that not necessarily needed yet?