Medical records could be revealed by AI training-data vulnerability

0:27If you felt stuck trying to lose weight, you're not alone. Ready to reach your goals? Visit ForHERS.com slash eligible to get personalized, affordable care that gets you. That's F-O-R-H-E-R-S dot com slash eligible. ForHERS.com slash eligible. Weight loss by HERS is not available in all 50 states. WeGovi is the registered trademark of Novo Nordisk AS. To get started and learn more, including important safety information, WeGovi clinical study information, and restrictions, visit ForHERS.com. Running a business means juggling a lot of moving parts. And when your communication tools can't keep up, things start to slip.

1:07Missed calls, slow replies, scattered conversations. They're not just frustrating, they're lost opportunities and revenue left on the table. That's where Quo comes in, spelled Q-U-O. Quo is the number one rated business phone system on G2, trusted by over 90 ,000 businesses. One shared business number for calls and texts. So every conversation stays visible, organized, and accountable. It works from an app or computer. You can keep your existing number, add teammates, and sync your CRM. Letting you scale without adding complexity. And with built-in AI, Quo logs calls, summarizes conversations, and flags next steps, even after hours.

1:49Stop missing customers. Stop leaving revenue on the table. Try Quo free and get 20 % off your first six months at quo.com slash tech. That's Q-U-O dot com slash tech. Quo. No missed calls. No missed customers.

2:22I find this not only refreshing, but at some level astounding. Nature.

2:34Welcome back to The Nature Podcast. This week, how medical information could be stolen from AI training data. And the strongest evidence yet that the universe is uneven. I'm Nick Patrickow. And I'm Benjamin Thompson.

2:56First up on the podcast this week, we've got a story about cyber attacks on medical AI systems. Attacks that could allow criminals to steal someone's personal information from the AI's underlying data sets. Now, AI systems are increasingly being developed, tested and used in medical contexts, for example, to quickly assess whether a skin lesion might be cancerous and needs to be checked by a clinician. These systems could be invaluable for aiding in diagnoses and helping patients access quicker care when needed. But these systems don't start off knowing what to look for. They need to learn from pre-existing images and information.

3:41And as Moritz Knoller from the Technical University of Munich in Germany explains, they're only as good as the data they're trained on. I think what has been shown to be the driving force for the success of these systems is really the availability of suitable data and very, very big amounts of it. What people find is that as you increase the amount of data and the amount of computation, these systems tend to get better and better and better. Specifically, these AI models are often trained on publicly available data sets of medical information that patients and health services have allowed to be used for research.

4:15These data are used to train the AI to form patterns and connections, but the use of this data can come with risks. As we move towards these systems that are trained on bigger and bigger data sets, what people are worried about is that when you make these models available that to some degree you might also expose the data that they were trained on. Moritz is part of a team who have a paper in Nature this week examining something called a membership influence attack or an MIA. MIAs are ways of probing the data used to train an AI. Moritz and the team wanted to see if feeding a piece of someone's medical information into an AI would reveal whether they were part of the data set used to train it.

5:00On the face of it this might not seem like a big problem. Well, let's say a criminal already has some, but not all of your medical information, and then puts it into an AI model to find out if this information has been included in the training data. It turns out that from this, they could learn a lot. We made this example with a model that's trained basically to predict, based on some blood data, so just general blood results, would this patient respond positively to immunotherapy or chemotherapy? And if I know that you were part of a training data set of chemotherapy patients, I now know that you have cancer.

5:38Something that I couldn't otherwise learn from your blood data alone. So I learned something new about you, and this is sensitive information potentially. Moritz wanted to work out how much of an issue MIAs might be to individuals in this kind of context. So he and his colleagues conducted their own attacks on a series of medical AI models. These had been trained using seven data sets of publicly available anonymized medical data, things like chest x-rays, skin images and mammograms. Different models were trained using different subsets of information. The team then fed the AIs a particular piece of data and tried to work out whether it was included in the training set or not.

6:20So what you look at is essentially just how the predicted probability changes based on whether a data point was included in a model's training data set or not. And how you do that is you simply query the model. So you send a request to the model, say, let's say with your chest x-ray, and you look at the prediction and check, is this prediction different when you are part of the data set or not? and you look at how does your predicted probability of having pneumonia change whether you were in a data set or not. It turns out that AIs are much more confident when making predictions about data they've seen before.

6:58Essentially, if you gave it a chest X-ray of someone with pneumonia and asked, does this person have pneumonia? You'd get a higher percentage chance if the AI had been trained using this image. And by using statistical methods, the team were often able to identify if a person had been included. The fact that you can do this almost perfectly, and we showed this across electrocardiograms, electronic health records, various types of medical images, could be reason for concern, right? Previous research in this area had assessed the likelihood of someone being identified in a data set by taking the average risk of all the people in the data set.

7:33Those results suggested that, on the whole, there was a minimal risk of identification by an MIA attack. Moritz and his colleagues' work didn't work on averages and instead looked at the risks to individuals. In their work, they found in some cases much higher risks. When a training data set was small, the proportion of folks in the data set who could be identified was relatively small, ranging from between 1 in 10 ,000 to 1 in 1 ,000, but on models trained using large databases, the number could be as high as 1 in 10. The team found that even if you only have partial data, like part of an EEG scan, MIA attacks are still possible, although with lower success rate.

8:16The team also investigated broader patterns to see if there were groups who were more likely to be at risk of being identified. So what we found is that small groups in these data sets tend to be overrepresented among the people that are most at risk. And conversely, on the other hand, if you look at majority groups, So people that make up most of the data set and they have very typical data, they tend to be underrepresented among the people that are most at risk. These smaller groups that are underrepresented in the training data could be because of things like race or sex, but also less tangible things like whether a person had health insurance, even how their medical image was taken.

8:58Perhaps they were sitting down rather than standing up, say. And although it's hard to say exactly what's going on, it seems that when there's only a few of a group in a data set, their records stand out statistically, and the AI remembers something about them. There is evidence that medical AIs can form connections and influences that humans can't see. In one example, an AI was able to accurately identify a person's self-reported race just by looking at an X-ray. And while this work was done in a research setting, Moritz thinks it is applicable to the real world and says that MIAs are already being attempted, so evaluating the risks is important.

9:40Marzia Kasimi from MIT in the US researches how to make medical AIs more equitable and safer for users. She was impressed with the work and its highlighting of potential risks for specific groups of people. I think it was such a good demonstration that average model evaluation hides risks for particular groups. And so I think the really nice demonstration here across such a large number of medical data sets and many different modalities, they're not just focusing on one type of data, show that these average attack performance metrics that can look like, like, oh no, on average we're not able to identify anyone, can hide that there's a subset of patients for whom membership can be inferred with almost perfect success.

10:28Marzia says that doing a broad evaluation of the risks that people who provide medical information may face is an important step, and something for other researchers to pick up on. What this paper hasn't done yet, but I don't think that that's a flaw, it wasn't the point, and it's opening the door for this, is give a recommendation for when and how we should be worried about these risks. So this really is a technical demonstration of a known flaw in machine learning systems, but it's not yet connected back into practice, which is now the task, I think, for more future work. Moritz says that there are methods already in existence that can, in essence, help to prevent any one group from statistically standing out in the training data.

11:15He also says that it's important to build privacy mechanisms like these into systems by design to ensure that people's data are safe, so they feel comfortable sharing it to train medical AIs. I think it's super important that people still contribute to these data sets, right? Because I think there's so much potential there to do amazing stuff and make huge progress on care and medical outcomes for people. I think from the perspective of a data contributor, right? Like I sometimes ask myself this question as well, right? Would I go ahead and contribute my personal data to train an AI model? And if you as a person have a safeguard and you have a verifiable privacy guarantee that nothing bad can happen because you contribute this data, this is a very powerful notion.

11:58Masia agrees that access to real data from real people is the way to make medical AI systems that could help clinicians and other healthcare professionals with their work. While she says that highlighting risks to people's private data is important, she also argues that this finding on its own shouldn't panic patient privacy advocates. While it does show MIAs could be used to infer data about someone, she suggests that there are more pressing and more straightforward ways to obtain someone's information. Nefarious ones like email phishing attacks and malware, but others too that are part of our daily lives, things like shopping data or social media.

12:37You know, the amount of data that's collected about consumers that also might indicate private medical conditions that they don't want to disclose is pretty vast. And so I think showing this attack should make people more aware, specifically model developers and model deployers. It should make them more aware of this specific fragility, which is something we're all feeling out about machine learning models as we develop them. I think patients should be aware this is one extra vector to think about. But more broadly, we should have laws on the books that protect consumers from all sorts of surveillance that might be used against them.

13:20Marzia Kasimi from MIT in the US there. You also heard from Moritz Knoller from the Technical University of Munich. To read Moritz's paper, look out for a link in the show notes. Coming up, new data suggests that matter in the universe isn't how our models say it should be. Right now though, it's time for the Research Highlights with Dan Fox.

13:46Most tropical butterflies survive in their colourful adult forms for just a few short weeks. But one group has evolved to live much longer. A feat which could give us insights into ageing. Researchers studied the Heliconius butterfly genus, which some 12 to 18 million years ago began adding pollen to the typical butterfly diet of nectar. Pollen provides the adults with nutrients that they can use to keep their bodies working longer and allows for prolonged reproduction. In fact, data collated from published field studies and public butterfly houses showed that at least one pollen-eating individual lived to the venerable age of 348 days.

14:28When the team removed pollen from the diet of one long-lived species, its average lifespan dropped from 63 to 47 days. 47 days is still significantly longer than most butterflies though, and this suggests that the Heliconius lineage has undergone evolutionary changes that support its protracted adulthood. That research is safely cocooned in Nature Communications if you want to learn more. A new sashimi bot can serve up a delicious seafood meal, using three arms and expert culinary skills to autonomously cut and arrange slices of raw fish. Seafood is slippery, limp and floppy, making it a difficult object for robots to manipulate.

15:14Researchers wondered whether they could design a robotic system that overcomes this challenge. Their solution has three robotic arms. One has a hand to manipulate tools such as a knife, one stabilises a salmon loin, and one wields a pair of chopsticks. The system learns through simulations using a process called deep reinforcement learning. Once deployed, the robot autonomously cut the salmon loin at various angles and picked up the resulting 34 sashimi slices which had a range of thicknesses. SashimiBot could help to produce this protein source more sustainably by processing seafood where it's caught.

15:50It also signifies progress in robots' ability to handle such challenging materials. That research is sliced, diced and served up for you over at MPJ Robotics.

16:07The universe doesn't work the way many physicists have assumed. It isn't evenly distributed. In general, in physics, the simple solution doesn't apply exactly to reality. This is Francesco Silos-Labini, a researcher who's co-authored a paper in Nature, which defies these assumptions. Currently, many models of the universe assume it's homogeneous, roughly made of the same stuff all the way through, and isotropic, which means it's the same in every direction. and this idea that the matter that makes up the universe isn't like this is kind of a big deal because physicists best model of how the universe works assumes that it is the main solution to the Einstein field equations which dictate how matter and energy are distributed in the universe assumes that everything is even in every direction and that is the basis of a lot of understanding of how the universe works.

17:09So the prima robber's work metric which is behind the solution that gives you the Big Bang is based on this assumption so this is why it is so fundamental. There has been lots of evidence that the universe is not in fact evenly distributed. Clusters of galaxies and voids where there's few of them have been observed. We even live in a galaxy that's part of a bigger cluster. But while these structures are enormous, they're only a small part of the overall universe. So the question is up to which scale? While there may be patches of the universe that have galaxies clumping together, it's entirely possible that when you look at the entirety of existence, things average out, and everything is, statistically speaking, homogeneous and isotropic, even.

18:00So Francesco and his co-author have looked at new data from the dark energy spectroscopic instrument DESI which gives the measurements of the distances and angles between millions of galaxies. Using a statistical measure they developed they could then figure out if things are as even as the models assume when you massively scale things up. Here's Francesco to explain how it all works. The first point is to select a sample which covers a contiguous volume. So there are no regions in which there is an incompleteness, or at least where incompleteness is large. Then once one has this sample, one has to count galaxies as a function of distance and angles.

18:46and see whether this counting is similar to what is expected from the models. Basically, what we did was to implement this statistical tool, which is called angular distribution of pie-wise distance, which counts the number of pairs as a function of distance and angles. And then we have compared the result with the result from cosmological simulations, which basically is the result of a model. what you expect from the standard model of galaxy formation. And by statistically mapping out the relationships between these galaxies, Francesco and his co-author found that galaxies clump into extraordinarily large structures, which don't conform to the assumptions of the models.

19:32As these structures were anisotropic, they were not uniform in every direction. So what we found is that there are anastropic structures which have a size of around 1 gigaparsec. 1 gigaparsec is one-four of what is believed to be the radius of the universe in the standard model. Part of the reason that these structures belie expectations is that, based on physicists' current main models of the universe's formation, they're too large to have formed in the time that the universe has been around. observations of the cosmic microwave background radiation the radiation that fills up the universe and can give researchers a snapshot of how it was in its early days suggests that around the time of the big bang everything was pretty homogenous there were some deviations from that which may have led to large structures but nothing on the scale of what francesco and his colleague observed here francesco believes that this is the strongest evidence we have yet that matter in the universe is not evenly distributed.

20:37And it lines up with lots of observational data that seem to point at the same answer. If that's the case, then physicists will need to rethink their models, something that may divide the community. I think that part of the community is already trying to develop models in which homogeneities are taken seriously, from a mathematical point of view in the solutions. So they will be very, very happy to have another piece of observation, which gives a motivation from a theoretical point of view to develop more sophisticated models. Others, of course, I mean, they will not be happy because, I mean, they have developed models in which the assumption of homogeneity in the axotropy is inside them.

21:25And so they will not be happy with that. but nature is not the ultimate word, let's say. That was Francesco Silas-Labini from the Enrico Fermi Centre in Rome, Italy. For more on that story, check out the show notes for some links. And that's all for this time. If you want to stay in touch, you can reach us on email. We're podcast at nature.com or we're on social media at naturepodcast. I'm Benjamin Thompson. And I'm Nick Peciciao. Thanks for listening.

22:24Thank you. slash shine26 for an exclusive offer of 20 % off your first order. That's Thrive Cosmetics. C-A-U-S-E-M-E-T-I-C-S dot com slash shine26. ACAST powers the world's best podcasts. Here's a show that we recommend. Do you want to know the best part about being married to a woman? That there's no man involved? I mean, true, but I was going to say that it's a sleepover every single night with your best friend. Oh yeah, that part's cute too. I'm Taryn, she's Cammie, we're married and staying up is our weekly pillow talk out loud with you. We're giggling we're gossiping, we're arguing classic marriage stuff.

23:05Just having fun being wives while we navigate growing up and building a family together. Then our sleepover grows. Our listeners call the PP hotline with their own gossip, burning questions late night spirals, all the stuff they'd only tell their best friends. So it's a private sleepover but you are invited Staying Up with Taryn and Cammie, new episodes weekly, follow wherever you listen Acast helps creators launch, grow, and monetize their podcasts everywhere. Acast.com.