Your phone can use tiny skin-colour changes to measure your heart rate

0:00Then I thought, what if I've scaled businesses, what if I've scaled my philanthropy? What if I did as much in one year as I've done in my whole life? See how your wealth could have even greater meaning at creativeplanning.com slash impact.

0:37Welcome back to The Nature Podcast. This week, how your smartphone's camera could measure your heart rate, and testing Richard Feynman's solution to the restaurant dilemma problem. I'm Benjamin Thompson.

0:59First up on the show this week, reporter Julie Gould has been finding out about a way that a person's mobile phone could be used to quickly and unobtrusively measure their heart rate. Resting heart rate, literally the number of times someone's heart beats in a minute while at rest, is a pretty simple number to calculate, but it carries a lot of information about a person's cardiovascular health. A declining resting heart rate over time can signal improving fitness, but a consistently elevated resting heart rate can be associated with cardiovascular disease, diabetes, and even early mortality. So knowing if this rate is increasing can be useful.

1:39Having the ability to be notified if you see trends are different, your resting heart rate is different, has been increasing over the last couple of months, that could trigger some further investigations as to why? Is it stress built up or is it something else going on in your health? This is Ming-Zo Po, a staff research scientist at Google. His goal is to simplify taking these measurements so that these warning signals can be picked up faster and, if need be, treated sooner. But clinical measurements of these numbers can be time-consuming. It requires dedicated time and effort to go to the doctor or to take a few minutes out of your day to rest, measure and make a note of it.

2:19There are devices already out there that make it easier, like smartwatches, which track resting heart rate without any effort from the user. But these devices cost money, require charging and need to be worn consistently. A large portion of the world's population, particularly in lower income countries, simply doesn't have access to them. But what if something that is ubiquitous could do the job? Well, this week, Ming is part of a team who have a paper out in Nature reporting a system that estimates a person's resting heart rate simply from brief videos captured while they're using their phone.

2:53It's called PHRM, or Passive Heart Rate Monitoring, and it uses machine learning to measure resting heart rate during everyday smartphone interactions without the need for using a specific app. PHRM is designed to run entirely on your smartphone device. and every time you unlock your phone to use it, to check email or to browse the web, it activates the front-facing camera to capture a very short video clip, about eight seconds. We train a deep neural network to analyze the video frames captured in this short video clip to infer your heart rate at that moment. And at the end of the day, we can aggregate all these intermittent readings throughout the day to estimate your resting heart rate.

3:36The technology being used by the front-facing camera is called remote photoplethysmography. Every time your heart beats, it generates this pulse wave that propagates through the blood vessels and changes the blood volume, including the vessels that are underneath the skin of your face. So blood absorbs specific wavelengths of light, primarily around the green-yellow spectrum. So these are the subtle colour changes that the front-facing camera or the smartphone camera is analysing to estimate your heart rate. Human eyes can't really pick up on these subtle colour shifts because the changes are on the order of something like 1 % of change of the baseline.

4:15But a digital sensor is able to by measuring the underlying raw pixel intensity and tracking the changes over time. Ming and his team then trained a neural network to estimate the heart rate from the videos using these minute colour changes brought about by pumping blood. But real-world smartphone use introduces huge challenges. People move, they walk or they sit in cars and lighting is constantly changing. So to address these challenges, the team built in automatic image stabilisation and cropping to ensure a person's face was the focus. And they also allowed the system to assess how good it thought each video was.

4:52The way we trained the deep neural network is instead of just allowing the network to output a single heart rate number, we also allowed the network to output a confidence estimate. So in other words, we allowed the network to express uncertainty. So if conditions were really bad and inconducive for a reliable heart rate measurement, the output confidence would be low, and the algorithm could then discard that unreliable measurement. We conducted a real-world study with participants using their personal phones in unconstrained conditions. And these two aspects are important, I'd say, both the personal phone use and the unconstrained use.

5:34In this part of the research, these two factors were important as they didn't want the participants to feel like they were being watched by the researchers. So in this experiment, the team collected more than 160 ,000 videos from 107 participants using 26 different smartphone models. And this gave them a wide variety of conditions to test their system on. Ming and his team then compared the passive heart rate measurements from the study to clinical standards, which he was able to meet when measuring daily resting heart rates passively. We set the target of achieving a mean absolute error of under five beats per minute, and the system was able to produce those accuracy targets across three skin tone groups in our study.

6:17But while the team was able to measure resting heart rates across all skin tones, the system still encountered issues. We deliberately tried to recruit and over-represent darker skin tone groups in this study, because one thing we also observed was that the measurement success rate, which we defined as the number of video measurements that returned a sufficiently high confidence level from the PHRM model to be deemed valid, was lower in the darker skin tone group. This meant that there's a lower number of valid high-rate measurements across the day for the Dr. Skinton group. So that's an area, I think, for further investigation and improvements.

6:59And there are other things to be considered too. The idea of a smartphone camera silently monitoring physiological signals obviously raises privacy concerns. So Ming and his team have recommended strong safeguards for the future use of the PHRM technology. The most important is making sure that informed consent is given, as well as PHRM only being run locally on the device and kept separate from the rest of the phone. You run it in an environment that the video frames are not accessible by other apps. So it's a safe, secure environment to execute the inference on. And we discuss tying it to face authentication so that the system would be smart enough to make sure that if you pass your phone to somebody else, it's not going to measure their heart rate and attribute it to you instead.

7:45While the team's results show that the system was able to measure a person's resting heart rate to a high accuracy, from a medical device perspective, they're careful not to overstate what these measurements can reveal. For example, the simple snapshot heart rate measurements cannot infer someone's psychological state. A spike in heart rate could mean many things. You're feeling a bit anxious, you've had some coffee, or you've gone for a short run. So the system is not ready for prime time yet, and there's still a lot of testing to do before it can be rolled out. But heart rate can be a useful indicator of potential health conditions.

8:19And for Ming, the broader goal is making health monitoring almost invisible and available to everyone. This was an example of the potential of what we could do with devices that we already own and how we can make health monitoring as seamless and invisible as possible. So I think if we do it in the right way, in the responsible way, I hope that, yeah, it can really make health tracking easy for everybody and make it accessible for everyone. That was Ming Zerpo from Google in the US there. To learn more about the heart monitoring system, head over to the show notes for a link to the paper. Coming up, should you stick to your normal order in a restaurant or should you try something new?

9:03Mathematics has the answer. Right now, though, it's time for the research highlights with Dan Fox.

9:13A rogue star seen streaking across the cosmos was probably hurled by the black hole at the centre of the Milky Way. Astronomers analysed light emitted from the star using the Dark Energy Spectroscopic Instrument and plotted its trajectory using position and velocity data from the Gaia Space Telescope. They found that this sun-like star is unusually rich in heavy elements, suggesting that it comes from an intense star-forming region such as the galactic centre. Given its trajectory and current velocity of 314 kilometres per second, the team suggests that it's likely to have been catapulted at high speed when its partner star was captured by the gravitational pull of the black hole.

9:56The authors say that this hypervelocity star could provide a way to study the chemistry of the inner galaxy, which is usually shrouded with dust. Get drawn into the gravitational pull of that research, published in Astronomy and Astrophysics. Atoms on the sparkling surface of solid gold spontaneously arrange themselves in ways that prevent oxygen from tarnishing the metal. Gold is not completely inert. Its atoms can form compounds of oxygen and other chemicals, but in its bulk form, the metal tends to avoid chemical reactions, and in particular the oxidization that helps to make copper green and iron rusty.

10:36To understand why gold never loses its luster, a team used supercomputers to assess the surface chemistry of gold crystals. These crystals are known to be composed of atoms arranged into cubes, but in some cases the atoms on the surface rearrange to form hexagonal patterns instead. The author's calculations showed that while square arrangements help oxygen molecules to split on contact, the first step to oxidisation, hexagonal patterns make that less likely, helping to keep the gold shiny. Findings, the authors say, that could help researchers to design gold-based catalysts. You can find that gold standard research in physical review letters.

11:18when i go and see my mum we often get food from a terrific restaurant that's around the corner i always order the same thing it has to be said vegetable bolty and sag paneer since you're asking and i love it but i do wonder if maybe i should be less of a creature of habit there's loads of good stuff on the menu so maybe i should switch it up and get something else it might be better, but then I might not like it as much. And it turns out there is a mathematical solution to this mouthwatering conundrum, and one that has quite a storied history. It's the subject of a new research paper this week.

11:56And joining me to talk about it is Nature's Davide Castelvecchi, who joins me now. Davide, hi. Hello, Ben. Thanks for having me. So I'm not alone, Davide, in this restaurant situation. Now, the story in this case starts in a Thai restaurant in 1970s California. Correct. In Glendale, California, which is a community right next to Pasadena, where the California Institute of Technology is located and where Richard Feynman, the legendary, some would say notorious Nobel Prize winning physicist, was teaching. He and a friend, they were regulars at this restaurant. And Feynman's friend, Ralph Leighton, wondered, you know, should I keep getting ginger chicken because I like it?

12:42Or should they be, you know, adventurous and explore the rest of the menu? And Feynman being Feynman, pulled out a notebook and started calculating. And soon enough, he had a mathematical solution. He had like formalized the problem into what we would call a decision theory problem. Right. And other people call this a stopping problem as well, right? Correct. And picking your best dish at the restaurant may not be the most consequential decision of your life, but it's a very similar problem that we encounter in many other situations in life. Like, for example, when you shop around for a new home or new place to rent, when do you stop?

13:24Like, at what point do you think you're satisfied enough as opposed to keep searching? And so Feynman then jotted down an answer to kind of figuring this out. And the story continues then, right? I mean, tell me about this solution that he wrote down. So the solution for many years was only known to Feynman because his handwriting was really difficult to read. But Ralph Leighton kept the notes. And many years later, in 2013, Tom Griffiths, who is a cognitive scientist at Princeton, heard about this and became interested in he actually went through the painstaking work of transcribing the whole set of notes.

14:05Wow. I mean, long time listeners to the podcast will know that my handwriting is absolutely atrocious as well. And my colleagues can't read it. And trying to figure out what someone means can be absolutely maddening. And was this pages and pages and pages of notes? Or was this a very short equation? What were we talking about? It was only a few pages. In fact, the solution is quite simple. But what Tom Griffiths subsequently did with two collaborators is analyze the mathematics and also generalize it. And they confirmed that Feynman's solution was kind of optimal solution, at least within certain assumptions.

14:39And not only that, they also recruited more than 2 ,000 volunteers to play basically an online game and test whether their choices would be consistent with this solution. And this is what this new paper then that came out this week is all about, right? It came out in PNAS. How did they go about testing this stick or twist problem. So they rephrased the problem a little bit. Instead of you visiting the same restaurant every evening, let's say you're visiting a new town, you spend a week there, and you have the option of visiting different restaurants every evening. You have your own rating for each restaurant every evening.

15:18And the goal of the game is at the end of the week to accumulate the highest possible score. Now, if your first evening you already hit a restaurant that is a 99 out of 100, then obviously your best solution is to keep going to the same restaurant every evening. But if you start with, say, a 50 on the first evening, then you're probably better off trying a different one the second evening. The odds that being adventurous pays off go down, however. So for example, if you haven't had better than a 60 on your fifth evening, then chances are that the last two evenings you are going to go worse. So you better stick to your 60 for the last two evenings.

16:04And could our listeners go to a restaurant with a calculator and try and work out their best options, Davide? What's this equation look like that these results show is kind of the correct one for this situation? Oh, yeah. In principle, you could imagine visiting a new town and having, say, seven evenings, seven choices to make. And it's a simple calculation. You know, you enter the number of remaining nights, you take square roots and you plug it into this very simple formula that tells you what score you have to get that evening to make you want to change. But of course, I guess there is the X factor because we have this idealized situation where you can work out your square roots, but sometimes you just get bored.

16:43Yeah, so this particular model doesn't take the boredom into account. I mean, you know, chances are if you visit a town for a week that you probably wouldn't get too bored if you order the same dish two nights in a row. And more broadly then, Davide, of course we have this unusual situation, a thought experiment which did have a mathematical solution. Does it have broader applications, do we think? Well, the field of decision theory and more broadly game theory is one that economists have been using in applications for a long time And it's been the subject of multiple Nobel Prizes for economics.

17:16And results like this, you could imagine marketing experts applying them to, say, for example, you have a restaurant chain and you want to figure out how often or for how long to have a special dish on the menu, you know, for a limited time only. Because maybe you have modeled people's behavior and figured maybe after a month they would get bored of it. And you've been reporting on this story for Nature. Davide, do you think it will change your ordering behavior when you go to your favorite restaurant, little bistro somewhere in London? That's a good question. I also tend to order the same dish at my favorite Indian restaurant here near our Nature offices in London.

17:59I don't know. If nothing else, I will probably keep thinking of Feynman and getting amused whenever I encounter this dilemma in front of a menu. Wonderful. Davide, we'll leave it there and we'll put a link to your news story in the show notes. Davide Castelvecchi, thank you so much for being here. Thank you very much, man. And that's all for this week. If you've enjoyed the show, do let us know. You can leave a review on your podcast app of choice or you can reach out to us on social media. We're at Nature Podcast. And we're on email too, podcast at nature.com. I'm Benjamin Thompson. See you next time.

18:39But I thought, what if I've scaled businesses? What if I scaled my philanthropy? What if I did as much in one year as I've done in my whole life? See how your wealth could have even greater meaning at creativeplanning.com slash impact.