Alertness on Screen: AI Model Trained to Detect Fatigue from Video Analysis

0:00So really what's happening here, I guess the headline is that researchers have developed a neural network to detect operator fatigue, and they're doing this using eye movement. So in a collaborative effort to enhance operational safety, researchers from St. Petersburg University and St. Petersburg Federal Research Center at the Russian Academy of Science have developed a comprehensive database of eye movement strategies. So this data actually aims to inform neural network modules designed to monitor the psychological states of individuals working in various critical roles, such as vehicle fleet drivers, air traffic controllers, and industrial plant operators.

0:36So the first thing I want to mention here is really when they're talking about this research of why they're doing it, they're like, look, this is for individuals in really critical roles where nothing can go wrong, right? Air traffic control. You cannot have an air traffic controller getting sleepy and, you know, missing the missing the mark. There's all sorts of terrible things that can happen. Now, at the beginning of this podcast, you know, in the intro, I alluded to the fact that this could be used for advertising in other areas. And while that may have been a little bit of a stretch, right, because really how this technology works is by monitoring your iris by monitoring your eye.

1:11I think that it's still not that far off that that could be a place where this gets to you know a lot of times you're on your phone using your phone and the camera is activated or you give camera access to TikTok or you give you know Instagram access to your camera you give all sorts of apps access to your camera that theoretically could be on and tracking you or maybe when you upload a video that's like a camera-based video to a social media platform TikTok, Instagram, Facebook, X, whatever, right? Like, let's say you, it knows that you actually like did the recording. So for example, on TikTok, you can be recording a video on TikTok, it sees your eyes on there.

1:51So like there is a case to be made that even like a lot of different apps that you wouldn't really think about or expect when you're uploading a TikTok video, TikTok could literally be tracking and detecting if they think you're tired at that very moment. So not saying that everyone's doing this, or this may be like a big, huge invasion of privacy thing that's immediately happening. But the fact that it's possible begs the, like, you have to discuss the implications of something that's possible because inevitably, like, either it's, whether it's a bad actor or someone claiming to do it for safety, like, it will get embedded into stuff.

2:21If it's possible, people will do it. So that's all I'm saying. I'm not saying that TikTok and Facebook are going to be spying on you, turning on your camera when you're not expecting it and seeing how tired you are or your laptop company. But also think about the fact that, you know, there is a, there's a big prevalent problem uh which is obviously made known by the fact that a lot of like modern laptops you buy today have like a a manual hard shutter to cover your camera because people hack into cameras all the time you can download a piece of software um that has access to your camera right like i mean theoretically something like zoom that needs access to your camera for video calling could also be hacked into and secretly access your camera when you don't know and a lot of people feel like this is a major evasion of privacy so they i've you know you can go on Amazon and buy these little slider window shutters that go on top of your camera lens on your laptop to like manually cover it with a piece of plastic.

3:12And this is all like so popular that to the point that a lot of really modern Windows laptops I've seen, some of the newest ones coming out, literally have like hard built in camera lens covers. So obviously they're doing this for a reason. People can hack into your camera. And so whether it's your phone or whether it's your laptop and you're worried about exploitation, like imagine if someone was hacking into your laptop, seeing that you like running this technology through your camera on you to see that you are tired. And then maybe that's when scammers go and try to give you a scam call because they know you're maybe you're more likely to fall for something or become a victim.

3:50If you are, you know, sleep deprived, and you're not functioning at your full potential. So there's the whole, there's the whole spiel about how this can be used for bad, I try to always give that for every AI advancement, but there's definitely some ways this could be used for good. So let's jump back into the research. The research team utilized a multifaceted approach. Essentially, they were capturing a range of behavioral and neuropsychological indicators to offer a more comprehensive understanding of operator state. So whether they are tired or alert, the research findings were published in the Scientific Journal Sensor.

4:22So Irina Soshina, who's the Director of Biological Sciences and the professor at the Institute of Cognitive Research at St. Petersburg berg state university emphasized the advantages of an integrated methodology she noted quote an integrated approach provides a more complete picture and a more objective assessment of the functional state so in contrast to approaches involving separate registrations of certain indicators that reflect the state of fatigue satosha pointed out uh really kind of the limited the limitations of current methods like cardiac time revival measures, which are often considered unreliable indicators of fatigue levels, right?

5:02That's currently what we use a lot of the time. And instead of kind of this research focused on a unique approach involving eye movement, which are believed to accurately portray the interplay between neural networks of static and dynamic vision and psychological indicators. So this is really, really interesting. They're literally looking at your eye movements, running a neural net, and they're able to predict how tired you are just from your eye movements, which is interesting because they don't need to see anything else about you, just your eyes. And if you think about it, your phone and your laptop camera are almost always going to be pointed in a place where they can see your eyes, whether they can see the rest of you or not.

5:41So, you know, anything that could get access to that camera could pretty much tell when you're tired or not tired, and they just have information on you based off of that. So I think the researchers have made their comprehensive database publicly available, and they're encouraging software developers to leverage the data to improve their own systems. They said, quote, we have developed a comprehensive database suited for training neural networks that classify a person's state of as tired or alert. That was Alex Kurniskev, and that's a research, a senior research associate in the Laboratory of Integrated Automation Systems at the St.

6:14Petersburg Federal Research Facility at the Russian Academy of Science. The data collected for the study was really robust incorporating various sensors including video camera, eye trackers, heart rate monitors, and electrocentophonographs. The operators were also assessed for other factors such as sleep quality fatigue and complex visual motor reactions. So the data collection took place at multiple times during the day, morning, afternoon and evening to capture a broad spectrum of operator conditions. And the research spanned eight days and included 10 participants engaging in a mix of passive and active tasks such as reading and playing Tetris.

6:54The entire process was video recorded for further analysis. I think that's a great move on their part. If people see holes in the way stuff was trained, you know, maybe they'll be like, well, it's just predicting that someone's tired because the lighting's darker or something. Although I'm sure it's just in a laboratory with the exact same lighting, but you know, that kind of thing. The whole thing has been recorded. And by adopting a multi-dimensional approach and making the data publicly available, I think the research holds the promise of advancing the field of neural network-based fatigue detection systems.

7:24So this in turn could significantly bolster safety protocols across various transport industries and defense sectors that's what they're saying in any way um i see something slightly different and i see the fact that if they're open sourcing this and giving it away to everyone great time for hackers and scammers to grab it and use it for bad but you know it'll inevitably hopefully be used for good as well so just take it with a grain of salt and know that this exists so keep that on your radar as there are some very interesting implications I believe with this technology.