How SDSC Uses AI to Transform Surgical Training and Practice - Ep. 241

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Our guest today is a machine learning and AI leader who's worked on projects ranging from sustainability and reforestation efforts to creating a virtual clothes swap platform for environmental family fashionistas. But for the past few and a half or so, she's been serving as Director of Machine Learning at the non-profit Surgical Data Science Collective, where she leads research focused on utilizing video data from surgeries to develop tools that can provide surgeons with immediate feedback and insights on their performance.

0:45She also recently gave a TEDx talk titled, Why You Want AI to Watch Your Surgery, which I encourage you all to go check out on YouTube after you listen to our conversation. Because she's here right now to talk with us about the potential for AI to help surgeons bring better health care to everyone. Margot Mason Forsyth, welcome and thank you so much for joining the NVIDIA AI podcast. I know. Thanks for having me. Margot, maybe we can start a little bit about your background. I alluded just a little bit in the intro. You've worked on various projects after studying machine learning and leveraging your skills and experience for a lot of AI, for what we'd call AI for good projects, but really just, in my view, projects that are helping, you know, improve quality of life for everyone.

1:28So maybe you can detail that a little bit and then kind of bring us up to the present with how you started, got involved with the Surgical Data Science Collective. Yeah, sure. So like you said, I've been working in the AI field for quite some time now. My first field of studies was computer science. So I've done enough software development. And then I realized that I wanted to do a bit more scientific projects. So I went back to finish my master's and specialized in computer vision. Computer vision is something that I really, really love because I'm a very visual person. And so analyzing images and videos is something that I'm pretty passionate about, I would say.

2:10And from that, I've worked on many different projects with very big video and image files. So like you said, I've worked on several projects that go from analyzing lumber scanning, for example, to satellite imagery to detect deforestation or surgical videos. So it's been quite a ride for sure. And I've learned a lot mostly on how to productorize AI and how we can use AI to make an impact in the world and have a focus on is it actually going to be useful, you know, and make a difference at some point. So that's kind of how I see my career so far. Have you found that the projects you've been drawn to, has it been kind of being drawn to the next sort of technical or scientific challenge, kind of pursuing the craft and kind of pushing the boundaries of what computer vision and image and video analysis can do?

3:09Or have you been driven more by the mission of these different projects or have you just kind of worked out that you've sort of been able to follow both of your blisses, so to speak? I would say both, actually, yes. Slightly luckily. yeah always i mean i've always been passionate about different sciences so i actually have a hard time focusing on only one thing or one project and when i learned about climate tech for example i really wanted to see how i could help and how ai could help process all of this gigantic satellite and imagery you know remote sensing is really hard to process and then i I learned about surgical videos and I learned that there were thousands of terabytes of surgical videos that were not used.

3:57And it's a pretty big challenge because surgical videos are really heavy, really long. You can imagine eight hours procedure. No one really wants to watch those videos. So that's when I was thinking that AI is indeed a perfect tool for this kind of data that is really big, really long, but also has a temporal element to it, which is quite difficult. And when I thought about that, it was a really interesting, impactful project. But also in terms of technical challenges, I thought it was interesting. And that's actually what made me join SDSC in the first place. So how did you find out? What drew you to these troves of surgical videos?

4:39I started to work at the Surgical Data Science Collective, STSC, when I met the founder, who is a pediatric neurosurgeon, Dr. Donojo. And he introduced me to this issue. You know, he was telling me I have all these videos saved in drives and I don't do anything with them. And I know many of my co-workers and friends and other surgeons have drives with hours and hours of surgical videos, and they are just sitting on their desk, not really doing anything with them. Right. Just to context set for the audience, for me too, is it standard procedure that surgeries are just all videotaped? Are these cameras that are inside of people sort of, you know, guiding the surgeons?

5:20Or are they sort of operating room overhead cameras? Or how does this, how does it work, surgical video? That's a great question. It's pretty diverse. We have a lot of endoscopic videos and microscopic videos. So endoscopic will go, for example, through the nose or any other part of the body where you need to see inside. And actually the endoscopic videos are a really good data point for us because we see the computer vision algorithm sees what the surgeon sees. And that is golden because there is a lot of information in these videos. For the microscopic videos, it is also used by the surgeons sometimes when they do the surgery to really magnify what they're looking at.

6:02For example, if they're operating on very small arteries, they need to have this big, intense zoom. That's what they're going to be using. Often it's in 3D for them, which we have some of those 3D videos as well. So yeah, it's a mix of microscopic surgical videos, endoscopic videos. And we don't have yet the video, you know, kind of like camera security from the OR, but some of our collaborators use this kind of videos to get a sense of what is happening in the operating room. Right. And so you met the founder. So the Surgical Data Science Collective was already in existence and you met the founder and got involved?

6:41Yes. It was pretty early on. So the Surgical Data Science Collective is a non-profit organization that was started by Dr. Donoho. And the main mission and the main idea was to create and analyze a repository of surgical videos in order to improve surgical techniques and patient outcomes. Because still today, there are 5 billion people who lack access to safe surgery. And there are at least 4.2 million people around the world who die within 30 days of surgery. So if we consider surgery as a disease, it would be the third leading cause of death. That's why, you know, the goal of STSC is to utilize these annual surgical videos to identify best practices, support medical education, or even predict potential outcomes and complications in advance of surgery.

7:39Right. So we've had other health care practitioners and people from the health industry on the podcast talking about the thing that comes to mind is talking about analyzing still images, you know, x-rays and scans and using AI to discover things often kind of, as you said, at that super zoomed in level that, you know, cancer prediction and that kind of thing. Tell us about some of the opportunities and challenges involved with analyzing all of the surgical video. I would assume, you know, the first challenge is just gathering the data and then processing all of it. But kind of take us through it.

8:13What is it that, you know, you said improving best practices and real-time feedback. So maybe you can speak to that as well. Yes. So the first challenge, like you said, is actually to gather all of this data. And like I said earlier, a lot of these surgical videos are stored on drives. And it's really difficult to get access to these drives. Sometimes, you know, you kind of have to go and fly somewhere and meet with the surgeons to be able to get the videos. Most of the time, the videos are not even recorded because people don't know what they can be used for. So why would they record them? So actually, one of the biggest challenge is asking people to press the record button.

8:53Right. I'm laughing, but I'm imagining, you know, if I was a surgeon, that'd probably be the last thing on my mind, right? So, yeah. Exactly. Yeah. I mean, that is definitely not the priority. And then if they think about recording, so pressing this button, they have to export the video from the device, walk around with a USB key, upload the videos on the laptop, upload to the cloud. So there are so many steps here for these people who are extremely busy, who have so many other important things to do of their day. It's definitely not a priority. So that is our first challenge. And that has been one of the biggest challenges that we've had.

9:30But we've been pretty successful in gathering at least a good first base of a surgical video library. By now, we are about 40 terabytes of surgical videos. Okay. And we expect to get more, you know. And the other challenge here is to get diverse surgical videos. We don't want, obviously, for an AI model, we don't want videos from one surgeon in one hospital doing the same procedure. Hundreds of hours of tonsillectomies is only going to get you so far, I'd imagine. Yes, exactly. So that is the other challenge, is how do we get these videos from diverse sources and diverse fields, which is also a lot of networking because you have to go and talk to the people and ask them to record and then do they want to work with us so that we can start gathering these videos.

10:23So this is the other second part of this challenge of data collection. But in terms of the other challenges, obviously surgical videos are quite long, but they are also temporal. So videos, right? So it is a different type of models that you would use for still images. We have the kind of the same architecture that you would use for other models, but we always have to think about the temporary of what is happening in the video. And that is actually how we implement most of our models. Let's say if you're trained to track surgical tools, you know, you have to think about all of the challenges that comes with that in surgical videos, which are going to be obstructions and can sometimes you have, you know, explosion of blood or something like that.

11:11And you want to be able to start track the tools without losing them or with dealing with these problems, which are pretty similar to other computer vision problems. But it is slightly more challenging because of how messy these environments are. Sure. I can only imagine. And so from a technical perspective, you know, obviously there are We're hearing all the time these days about video models in the news kind of being opened up for consumer use, that kind of thing. You've been working with the Data Science Collective for going on two years now, is that right? Yes. So are you using, are you building tools yourself?

11:52Are you using off-the-shelf tools, kind of modifying them to suit? Do you have partnerships with other AI labs? How are you kind of fine-tuning the tools to get what you need out of them? A mix of all of what you just said. We usually will. So, you know, we're a pretty small team and a nonprofit organization. So we will try to use the most efficient methods for us. A lot of the times that we'll be reusing some architectures that are existing and then fine tuning them to our needs. combining some architectures is something that we've done a lot especially with the temporal model so having you know a mix of a cnn and a temporal architecture or we've been playing a lot with vision transformers and more recently with vision text transformers which are the big models you're talking about here and a lot of the time we will always be careful about new technologies So we want to try them and we want to make sure that we stay on top of the innovation that is happening to see if we can apply to the surgical data science field.

12:58Something that is quite interesting and challenging with this kind of data is that it requires a lot of expertise that as computer scientists, engineers, we don't have. So we need to work very closely with clinicians and surgical experts. And that's where the most important part of the work is happening, actually. not even the model architecture or the new cool AI tools. For us, it's really understanding what the expertise is and then what models should apply to bring that information that will be useful to the surgeons. Can you maybe walk us through an example of, and correct me if I'm wrong here, but imagine you have partnerships with surgeons and other medical professionals and institutions, and are you sending them images or videos to just kind of analyze and let you know what they see or kind of what's the, I guess I'm wondering what the process is or what it's like getting from footage and people revealing the footage to then an outcome that other practitioners can benefit from, whether it's, you know, a new technique or refining a best practice or something like that.

14:08So for most of our collaborations, we will work with clinicians and surgeons who have videos, but they don't have the computer science knowledge. So they will come to us to do all of the computer vision and analysis. So when they start working with us on these projects, maybe I can go through a concrete example. We've been working with several NGOs who are focused on surgical education. One of them is called AllSafe, and they focus on teaching surgical procedures to several students all across low-income countries. And they do it through a digital platform. So it's online courses, and then the review is done through videos.

14:51And so what we're trying to see here is, can we analyze these videos and give feedback to the students with computer vision? And that is useful. So that is the important point, is that is useful. So then we will work on developing the computer vision models to extract the features that we need to be extracted to do the analysis and then collaborate with the clinicians and the surgeons on what exactly do they need to have in the feedback or what do they believe is something we should focus on. because most of the time, you know, we're going to look at something and I'm going to think with my engineer mind, oh, I'm going to look at this feature and I'm going to make this graph and it's going to be amazing.

15:33And then I say to the surgeons and they're like, what? What is that? So that's why it's called the Sturgical Data Science Collective because the first step is creating a community with the clinicians and the computer science experts. And we also have some collaboration with computer scientist groups, so where we will work with them to analyze some of the videos we have. So that is almost a connection between we have surgeons who want to do something very specific and we have computer scientist collaborators who can help us do that specific task that maybe we don't have bandwidth for. So we are trying to expand that part of our community as well to really, you know, have a real impact and scale that because it's not only going to be us, it's going to be a whole community effort.

16:25I'm speaking with Marder Mason Forsyth. Margot is the Director of Machine Learning at the Surgical Data Science Collective, a nonprofit that is using AI machine learning tools to analyze video data from surgeries to develop tools and feedback loops and other mechanisms that can help surgeons with insights and feedback on their procedures and techniques and really just bring better health care to more people across the globe. as Margot was just talking about. You mentioned, you know, being a nonprofit doing AI research is a little bit unusual right now. What is that like? Are there big things that are either, you know, well, I mean, obviously I would imagine funding resources is an issue as it is for almost all nonprofits.

17:11But are there things specific to being a nonprofit AI kind of research group that stick out to you? So there are many interesting aspects that come from being a nonprofit. Like you said, resources are indeed limited, so we have to be creative in the way we train computer vision models. We will always start simple, which is actually something I've always done and advocated for, is if you want to start a computer vision project, maybe you don't need to start with the biggest model that exists. You know, start simple with a small data set, do a proof of concept, and then iterate. So that is what we have as a development pipeline and research pipeline.

17:54We will always start simple and small and then scale. And that is limited because of our resources, obviously. But to me, it's something that is actually good. I would do the same if I had, you know, 10x budget. I would probably do the same, but it helps in that way. And then it brings a lot of different projects. being a non-profit we were able to work on projects that maybe we wouldn't be able to work on if we were a for-profit for sure it would be it would actually be completely different and that's why i really wanted to give sdse a shot when when i first met dr denoho because i was curious about i was like how are we gonna how are we gonna do that you know i've never done ai i've never seen ai done in a non-profit there are some others but sure it's really research-focused and community-focused, which you wouldn't really be able to do as well in a for-profit, I believe.

18:51I'm going to ask this and answer, please, based on what's happened so far and or, you know, what you see coming in the near future. What are some of the big benefits for clinicians, for patients that you've seen or expect to see from, you know, not just the work that you're doing at The Collective, but more broadly leveraging AI to help with the surgical process? The AI field will bring a lot of new and good things to the medical field, I believe. In the surgical space, which is what I've been exposed to mostly, it will bring a lot of standardization. Something I've discovered working in that field is that every surgeon in every hospital will perform surgeries and procedures in different ways.

19:35And no one really knows A, B, C, D of how you're supposed to do a specific procedure. So by having a tool here, AI, to first encourage people to collect the data. So first, we're going to get the surgical videos. We're going to finally start looking at these videos that are not being looked at and then share them between surgeons all across the globe. That will bring a lot of standardization or at least they will start to talk to each other, which I think is kind of beautiful because right now you don't really have a good way to talk to each other. And through the surgical videos, the hope is that they will start to talk to each other.

20:14And when you can imagine so many applications, for example, I mean, the one that always comes back is education. Instead of using a medical textbook with drawings, the students can watch a tutorial on how to do a specific procedure. So that's a big difference that I think will change a lot of things. and then being able to find best practices through this analysis of surgical videos is going to be pretty interesting because who knows what is in these videos. And there's so much that has to be discovered. And there is a big need to be creative when we think about this data because no one has ever looked at this data and no one has ever really thought about what can we do with all of that?

20:59And what is my question that I want to be answered? Right. And that's one of our challenges, actually, is sometimes we ask surgeons, oh, what do you want to answer through all of these videos that you have? And they don't really know because they haven't had this option before. Right. That's interesting. It makes me think, as I mentioned, there were some of the examples of MRI and scan analysis and cardiac care and that kind of thing. And I'm thinking about the AI tools being able to help practitioners find, you know, differences in cells on a very, very sort of nano basis. Right. But even with that, I'm thinking, oh, well, they know what they're looking for.

21:40Or even if it's they're looking for an anomaly, it's still kind of we know what we're looking for. But yeah, with surgery, my very kind of naive, you know, not knowing much about the field coming in this conversation thinking, oh, well, you know, video footage is being used to train AI systems. So are we moving towards, you know, better education for humans or even training robotic surgical arms or that kind of thing? But it's fascinating to hear you say that it makes sense to me as a non-surgeon that what would they be looking for? It's, you know, it's not the same as looking for, you know, an anomaly in a cell that might stick out.

22:13I think at the beginning, probably when they first started to analyze MRI with AI, they also had to be creative because someone had to be asking for these questions. And for surgical videos, one of the first steps would be to look at anomalies, which actually we're starting to do now is what are the outliers? Who is using this tool and no one else is using it for the same procedure? So we are kind of starting with the low hanging fruits, I guess, but the deeper existential questions are not there yet. And I'm really excited to work with the clinicians to help them come up with these questions by showing them the data because no one else is going to come up with these questions.

22:54It has to be the people who are working every day in the OR. And actually, the videos are a really great source of data, but there is so much more going on. And obviously there is, you know, the patient data, there's the patient outcomes, there is everything that is going on in the operating room. And all our engineers have actually been in the operating room so that they understand what is happening behind that camera. And I've been in the operating room too a couple of times now, and it's really helped me understand better what is happening. And sometimes when we have a new procedure type that we're exposed to, I go to the OR because I want to understand better, like, oh, some random questions sometimes like, where are you?

23:36Where is it in the body? Or how many people are operating? Because sometimes you have more than one surgeon. It's just so many things that you don't capture in the video, but there's still obviously a lot of information in the videos. Fantastic. I go for listeners who would like to learn more, or hopefully, perhaps, there's even some surgeons, some clinicians listening who are thinking, oh, I have surgical video that, you know, in a shelf on a drive somewhere, maybe I can send it and help out the cause. Where can listeners go to find out more about the work that the Surgical Data Science Collective is doing, the work that you're doing, perhaps to get involved as a partner?

24:13Who knows? Where can listeners go to learn more? So we have our website is thesurgicalvideo.io. And you can also find us on social media at surgical.sciencecollective. And I would also encourage if anyone is a computer engineer, computer scientist who wants to work on the different project that they've been working on and are interested in surgical AI to also reach out to us because we are working with quite a lot of different parts in this. So anyone who's interested should reach out to us. Fantastic. Well, Margot, thank you so much for taking the time to stop by, join the podcast and talk about the work you're doing.

24:49It's, I don't know, stories like this where the technical aspects kind of match up with the societal impact, I think, are just fantastic stories. They're sort of something for everybody, right? And it sounds like you're finding a really interesting path to fuse your technical interests with making an impact in your own work. So congratulations and all the best of luck to you and all of your partners and cohorts at The Collective. Well, thanks, Shinwa. Thanks for having me on the podcast. I really enjoyed the conversation. Me too. Our pleasure.

25:58¶¶