Secure Communications in Embedded Systems with Ismael Valenzuela and John Wall

0:00BlackBerry is a Canadian company known for its pivotal role in the smartphone market during the 2000s. Today, BlackBerry has adopted a major focus on cybersecurity. John Wall is the SVP and head of BlackBerry QNX, overseeing engineering, product, and operations. Ismail Valenzuela is the Vice President of Threat Research and Intelligence at BlackBerry, where he leads threat research, intelligence, and defensive innovation. John and Ismail join the podcast to talk about cybersecurity at BlackBerry, including secure communications and embedded systems. Gregor Vand is a security-focused technologist and is the founder and CTO of MailPass.

0:39Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at Vand.hk.

1:05Hi, John and Ishmael. Welcome to Software Engineering Daily. Hello. Thank you, Gregor, for having us. Yeah, great to be here. Great to have you both here, both from BlackBerry, which is a company that I'm sure our listener base definitely know of. But I also suspect that quite a few listeners maybe don't know of it beyond the handset business. And I'm sure many had handsets back in the day, like myself, a product that I love dearly. And, you know, today we're not here to sort of rehash the history of BlackBerry. You know, I read the book Losing the Signal. It's a very good book if the history lesson is what someone wants to dive into.

1:43And, you know, it inspired a film as well. So that's where we'll kind of leave that. Today we're going to be speaking about the QNX platform. And we're also going to be speaking about threat intelligence and cybersecurity at BlackBerry, which is a huge part of the company offering today. We're going to start with you, John. You have quite a deep history with the QNX side of the business. Love to just get a bit of a history actually on you first. Like what was your journey through? I mean, I think QNX is quite, excuse the pun, embedded in your work history. Absolutely. And then, yeah, how has that sort of led into QNX today?

2:17Yeah, I mean, some will say I came with the building. So I've been at QNX since 1993 when QNX was a private company building embedded software running on x86 PCs at the time, really focused on industrial automation and factory control and medical devices. Then towards, you know, as we got closer to the 2000s, we started getting into automotive. We had Delphi as our first customer. So Delphi being a part of GM at that time spun out and we were doing systems for GM. And there's a little company in Germany that noticed what we were doing called Becker that happened to be owned by Harman. Harman, when you think of JBL, Harman Kardon, et cetera.

3:03And the biggest part of the Harman portfolio was the Becker Automotive Group in Germany. And they bought us and we really started getting heavily into automotive. I personally was the person that was nominated to interface to the parent company. So I built a team. That team was there to serve the parent company in all their programs. Very, very difficult. Lots of automotive programs, infotainment programs, a lot of pressure, seven days a week, 14-hour days. Regular trips to Germany, a lot of yelling, getting screamed at by automakers. But it was an amazing journey. It taught us a lot. And then in 2010, we got sold to BlackBerry.

3:48And it was very interesting because it was a technology buy, wasn't for revenue or for anything else. It was strictly a technology buy. And the idea was for the QNX operating system to become the foundation of the BB10 phone. Interestingly enough, while I was managing the group that was interfacing to our parent company, once we got bought by BlackBerry, the kind of independent part of QNX started working for BlackBerry on building the handset. And I kind of stayed back and said, no, I want to continue this automotive thing that we're doing. I want to continue to sell QNX as an independent product.

4:26And we kind of rebuilt QNX from that point on. And you could imagine in those days, 2010, BlackBerry was still a huge company. QNX was a rounding error in comparison to the amount of revenue that was coming in from the handsets. So I was able to keep my head down, kind of operate it out of sight. and start to rebuild QNX. And we really started focusing on not just infotainment, but where the cars were going with autonomous drive, focused on safety software. And obviously, as the company's fortunes in handsets started to diminish, QNX started playing a bigger and bigger role. As we got more successful in automotive, as cars and other devices that they became more software defined, especially when you're thinking about mission-critical software, mission-critical devices that is what our products serve, we started to grow within BlackBerry and become less of a rounding error and a little bit more important to the revenue of the company.

5:31And that brings us to where we are today, where we feel that we have a very good future ahead of us and we play, I think, an important role within BlackBerry. Yeah, very interesting. We're going to talk a lot more about the automotive out of CIDA and some other industries. Just flipping back very briefly to your history, have you always been in embedded systems? Was QNX a sort of accident almost to get involved with, or what was the story there? Yeah, I mean, I graduated in 1992. I'm an electrical engineer. We were in a recession. I had a friend that worked at QNX. I never intended to get into software.

6:06I was more of a hardware person, an analog person. Started working at QNX. You know, obviously I started at the very bottom. I started in tech support, which believe it or not in those days was a phone call or a fax. There was no email. So we were dealing with the customers directly. And I really enjoyed the customer facing part of it. And so I tended to gravitate. I moved up within the organization from an engineering perspective, from tech support to an engineer, to a development manager, to a director. But I always kind of stayed on the side that was more customer facing, you know, engineering services, the groups that tended to have to deliver to the customers.

6:47So that was something that I found very fascinating. And I really enjoyed having an outward view as opposed to more of an internal view. And so I always gravitated towards the jobs that allowed me to be in front of the customer. Yeah, very interesting. Yeah, talking about QNX, let's stick on automotive sort of briefly. I mean, there's other industries that I believe we can also talk about. I guess it's just sort of trying to set the platform in context. Could you give some examples? You've mentioned obviously JBL, Harman Kardon, that might give some hints as to what QNX can help with. But what systems are we talking about and why do we need such a sort of, I guess, specific or proprietary embedded system to run these things?

7:27And let's just stick on automotive for now. Yeah, maybe a little bit of a history lesson there and maybe to talk about the progression. When we first got into automotive, it was infotainment. Android did not exist. iOS did not exist. QNX had a very rich environment for infotainment systems. We had multimedia engines. Our founder at that time was fascinated by becoming the next desktop and competing with Microsoft, Windows 95, et cetera. So we had all the elements needed to build infotainment systems, and we became the dominant force with well over 60 % market share of infotainment from 2004 to 2012.

8:13So that's really where we cut our teeth in automotive. So if you were driving an Audi or a BMW or a Porsche or a GM vehicle or a Chrysler vehicle in those days, the infotainment system would have been based on QNX. So from anywhere from 2008 to 2018, most of the vehicles that they were running QNX infotainment systems. So your navigation, your multimedia, your iPod integration, your iPhone integration, Android. But then in around probably 2013, we started to see that Android was coming on and trying to recreate the handset experience every single time with every single OEM was not going to be possible.

8:57And that they were going to take Android, you know, it was a full platform. It had everything you needed for the experience within the vehicle. So then we started to look at, well, where is the car going? And what we could see even in 2014 is if autonomous drive is going to be a thing, it's going to require a lot of CPU. It's going to require a lot of processing power. So it's going to require a high-level operating system like the type of operating system that we have. It's going to require real-time operating system, so deterministic operating system. And above all, it's going to require a level of safety certification that is very difficult to achieve and very unique to niche players.

9:39And so we really put our heads down, really focused on that. And so today, what you'll see us in, if you look at Mercedes that announced the level three drive system that they have in California and in Germany, that's a QNX based system. If you look at BMW that announced a level three drive system, that's QNX as well. And if you look at probably most of the cars out there, I think our stat is 24 of 25 EVs and all the other cars, any safety based system that requires a high level operating system, i.e. is running on a high performance compute, is running QNX. So for instance, if you look at NVIDIA that have the NVIDIA drive, the operating system is QNX.

10:24If you look at the Qualcomm solution, Qualcomm Ride, it's running on QNX as well as an operating system. So just about every advanced driver system out there today is based on our software. That's fascinating. So I think it's fair to say that most of our listeners will have interacted with a QNX-based system at some point. They absolutely will. And the other area that is very popular is what we call a digital cockpit. So you used to have your infotainment system. Now what they've done is they've consolidated the infotainment system with your digital instrument cluster, your HVAC, used a hypervisor virtualization solution to be able to run Android and to be able to run maybe some safety systems like the cluster.

11:06So we are dominant in that as well. So, I mean, there's obviously quite a few areas to unpack here in terms of how this all kind of works, I guess, under the hood. I'm aware that there's this idea of microkernel architecture, I believe. And I can only imagine that sort of leads into quite a few areas, performance, also just the real-time capabilities, etc. Could you speak a bit to what is that microkernel architecture, why, and what does that sort of make possible? I think the why is the history of our founders. is that the university project that they did was a microkernel approach. And just to give your listeners a little bit of knowledge of what this means is in a microkernel architecture, it means that everything runs in user space.

11:49So your drivers, your applications, they all run in user space. They can be stopped. They can be started. Whereas with a monolithic kernel, you're talking about something like Linux or Windows, where everything is kind of linked into one blob, sharing the same address space. So the advantage that that gives us is it gives us a big advantage on reliability. One application or one driver does not take down the system. It also makes the approach of safety certification much easier for us based on the architecture. So it really allows us to be able to update systems in the field that have safety requirements without having to redo a whole recertification of the system.

12:33The disadvantage of a microkernel is you have to be much more careful about how the system is designed to achieve performance because you are in different address spaces. There's context switching as you go from one application to another. Your drivers are all separate, whereas in Linux, they're all in one space, which makes it very efficient from a performance perspective. But we have all kinds of ways of mitigating that. But it really gives us an advantage for making the system robust and also for making the system, I would say, self-healing. So that if something does go down, you're able to catch it.

13:10You're able to restart those applications without bringing the whole system down. Yeah. I mean, so just to, I guess, give a very layman example, being able to reboot just the nav system without touching. Yes, exactly. Exactly right. Control and that kind of thing. Yeah. And have the rest of the system continue to run. Yeah. Yeah, it's probably fairly clear, you know, I think to our listeners listening, you know, in terms of why that was so important, as well as I'm sure, maybe older cars that they've run, and there's been some aspect of the system that's stopped working, but ultimately, it's not the whole system.

13:41Is it fair to say, I mean, again, being sort of software engineering daily, you know, the analogy of your microservices versus monolithic? Is that a good analogy? as well. You know what? That's a very good analogy, actually. That's exactly how we would look at it. You can add services, you can take services away, you can restart services. Yeah, that's a very good way. So from a software update perspective, you can update the system without having to take down or rebuild the kernel. And that avoids, like for instance, if I want to keep the iPod driver or the iPhone interface fresh, I don't have to update the entire operating system i can just update that particular piece of the system so that's a good analogy awesome so yeah i mean there's a couple of areas i mean talk about i think it's called qnx accelerate which is sort of a cloud piece to this as well could you speak to that because i'm curious how does cloud now come into this so to be clear our approach with everything is cloud first so a big initiative that we have you know one of the pain points for our customers and not just automotive, but across the board is vendor lock-in on hardware.

14:49So we take very seriously the idea of standardizations that separate hardware from software. So our operating system is fully POSIX compliant. We have made a lot of investments in something called VerdiO, which is shared interfaces to shared drivers. We've worked, for instance, with Google very closely to have a design where you can take Android from Google and drop it on any hardware without any changes. Whereas typically, if you're getting Android, you're getting it from the hardware vendor. You're getting a version from this hardware vendor because they're making all kinds of adaptations. So one of the ways to really force that issue is you do cloud first, where you have to abstract.

15:31There is no dedicated hardware necessarily. In a lot of cases, you're doing either an emulation or you're doing a software implementation. And the idea is cloud first, start your development in cloud, even if the hardware is not available, even if you haven't made your decisions on which hardware to select. We have all the hardware that is typical for those different domains that we're dealing with already supported with that platform. So you can develop in the cloud. And there's so many advantages of developing in the cloud, you know, keeping your tools straight, keeping your software versions straight.

16:06When you look at some of these really large software programs at automotive customers that you know they're around the clock around the world there's teams in india there's teams in china there's teams everywhere the biggest problem is keeping everybody in sync across the world and that's you know cloud removes a lot of that complexity so that's very important for us so yeah i mean in terms of devx effectively does does qnx have its sort of i don't know its own ive or how does it look when a developer is wanting to interact with QNX? That's a great question because this is something that we're being very careful about.

16:40We are not creating a CICD environment. Our customers have their own CICD environments, very complex CICD environments. So what we do is, obviously, we have our own tooling. We have an ID, VS Code. So we've just moved away from Eclipse to VS Code. And a lot of the reason was for cloud. But we're really focused on how do we provide microservices that can plug in to the customer's CICD to provide artifacts like SKU management, safety artifacts, S-BOM. The software bill of material is a huge topic these days. Cars can't ship in Europe if they don't have a proper bill of material. So while we don't provide an end-to-end CICD, we do strive to plug into them in a generic way.

17:25And we're working with the super scalers to do that, as well as our customers. and i'm aware i believe it's qnx 8.0 is the latest and greatest could you speak a bit to just like what does that kind of brought over i think 7.1 was the last for yes correct so 7.1 was based on the kernel that we had developed in the early 2000s so was developed with smp in mind symmetrical multi-processing but at the time when we were doing that especially in the networking space, you were talking about two distinct PowerPC chips with a bridge, and we scaled really well with two cores. We're now in the area that it's not unusual for even a mid-level processor to have cores.

18:15So we were running into a situation where you would have a deadlock on the kernel because you had all these cores that were trying to do kernel messaging. So the big change that we did with SDP-8 is we redeveloped our kernel for the first time in 20 years, 20 some years. We are now have the ability to scale one-to-one as a number of cores increase. So whether it's 16, 32, you know, our target was to match Linux on performance. So we wanted to retain the pedigree of safety, the pedigree of security, the pedigree of determinism, real-time response, but be able to match Linux on performance. And that's what we've been able to do with SDP8.

18:58So it's a monumental change for us from 7.1. And it's the foundational product that will carry us into the next 10 to 15 years. Wow. And we're going to move on to more in the security space, like pure security shortly with yourself, Ishmael. Just kind of wrapping up for now on QNX. I mean, I imagine just for, I guess, from what you've said, 8.0 being like a pretty major rewrite, you know, security must have come into that you know things have changed dramatically and not to say obviously 7.1 obviously was trying to keep pace i imagine with the threats of today but just in terms of it's a very specific context of how security can affect you know a vehicle or you know qnex is in medical devices and this kind of thing what are the security considerations and again maybe what does 8.0 do differently there so 7.1 was very focused on security that was the difference between the previous versions and seven.

19:54So, you know, we had put in a lot of gear for security. I think what's different with eight is there's now standards around security 21, 4, 3, 4. There's WP 29, 155, 156. We're now certified for security the same way we're certified for safety. So I think more than mechanisms, I would say it's more processes that have evolved to consider security more at the same level as we've treated safety. And it's a heavy lift. I mean, there's a lot of work that goes along with this. Obviously, we've added more mechanisms to SDP-8 as well from a security perspective, but 7.1 was quite good. It was quite good.

20:39It had all the mechanisms you would expect. we also have a lot of third-party partners that we work with that do like binary sealing they add a lot of things that even on top of that but i think the biggest change has been the process that we've instituted where we now look at security as being the same thing as safety from a process perspective fascinating okay thanks so much john for all of that on qnx and we're going to move slightly sideways or diagonally to Ishmael. And you are VP of threat intelligence, is that right? Threat research and threat intelligence. Fantastic. So yeah, I mean, again, for our listeners, BlackBerry is today is quite a sort of fairly sprawling company.

21:21It does have a sort of a few arms in different areas. I'd love to just, you know, first of all, again, a bit of your background, you have quite a illustrious background from a security standpoint. And then sort of how did that lead into BlackBerry? Well, just like John, we've been doing this for quite some time. I started doing, well, cybersecurity. We call it cybersecurity now, right? But we call it back then information security at the end of 2000, 2001. And yeah, I've been doing this for quite some time as a practitioner, doing hands-on work, working as an incident responder, as a consultant, walking into large environments where on fire, when the adversaries were there, maybe ransoming the environment more recently, or back then it was just like botnets or directed targeted attacks from different threat actors, including nation states, and then, well, helping these customers save the day.

22:14And then more on the proactive side, defending organizations, building security operation centers. And for the last few years, more on the research, engineering, supporting data science and engineering in building products for defenders, right? That's what I like to think myself of as just a defender that is trying to help organizations. Nice. And, you know, I think you joined BlackBerry almost three years ago. And, you know, if one just, let's just say, goes to the BlackBerry website, now security is just everywhere in terms of that's the sort of almost the face of the business now. If there's one thing it should be known for, it's security.

22:48So what does your role sort of encompass? And I'm curious if it does stretch across into anything to do with QNX, but broadly speaking, what does it sort of deal with day to day? So as you just said, right, the history the company has, the foundation in protecting devices is something that we have continued, right? John talked about how we have continued that on the automobile side, but also many other type of endpoints, right? Medical devices and other type of endpoints. And we continue to protect mobile phones as well, right? I mean, the audience cannot see my phone right now, but I'm holding it.

23:19And if I open my phone, I can just go on to the BlackBerry UEM, which is the Unified Endpoint Management, which protects my data here, my corporate information. We have software to protect communications with military-grade encryption. And threat intelligence, which is essentially what my team does with the research part, it's about learning, studying what adversaries are doing to try to anticipate their moves and to translate that into what we call actionable countermeasures. or defenses, right, that we can implement in the products and that we can also implement in our services because technology is just part of the problem.

23:56It's also about humans, human beings, understanding the context, the business. And if we take it to the safety side, right, this is all very related to each other. We also have software that helps to protect people in times of crisis because secure communication is also essential in critical event management. One of the kind of key products, Silance and Silance MDR, and that was, I believe, an acquisition. And back then, AI was being talked about as one of the key drivers, I believe, for bringing that into the portfolio. And I'm curious, how have you seen things evolve from then to now in terms of the usage or the application, I guess, of AI?

24:39Plus, I guess, the work that you do, which is, if you want to call it almost pure human, sort of understanding threat intelligence. And then how do we translate that into something where AI ends up sort of doing the heavy lifting? Good question. There's a lot to entangle there. But yes, BlackBerry bought Silence in 2019. And I remember back well before that, Silence was known for being the first technology endpoint solution to essentially focus on detection based on AI and machine learning models more specifically. And I remember back in the day, you know, 2017, 2018, people at Black Hat like, you know, laughing about these things and saying, oh, you know, that's not how you do detection on the endpoint.

25:18Fast forward to not even 2024, but even during the last few years, there's no vendor, no one that would dare to say that they're not using AI and machine learning in order to scale your detection capabilities because there's no way to do this when attackers are throwing at us, you know, a lot of malware per second as we report in our research. So that's a fundamental aspect of that. And obviously we have continued to build on this platform, adding more machine learning models. I think we have released up to 18 new machine learning models in the last 18 months. for various things that we see as a result of our research, working together with engineering and data science.

26:01We evaluate these things in realistic scenarios. So we do this, as I said before, in a proactive manner, trying to anticipate it, doing purple teaming exercises, which is essentially a way of emulating an adversary and having the blue team, the defenders, trying to catch these adversaries and trying to prevent these attack chains as soon as possible. So we do a combination, obviously, of AI, machine learning models with any other effective way of stopping these adversaries, including humans, right? Human beings that are monitoring and reacting to these type of alerts. There's like this silence, MDR, which managed detection and response, and I think just pure endpoint as well.

26:41And so is that all part of the same product portfolio or MDR, that's where the human side, sort of they come and actually do a lot of, eventually have a team at BlackBerry kind of helping you or how would you sort of explain the difference there? Yeah, so it's part of our portfolio, right? If you look at our website, we have a silence MDR, that's the human team or SOC that is 24-7 reacting to these type of alerts, being proactive, working with the customer and making sure that they have the necessary defenses in place to be able to take the most out of this. And for example, right now, we back this up with a$1 million guarantee for all of our customers.

27:19And then the rest of the products, right, the technology, silence endpoint, As I mentioned before, BlackBerry UEM to manage devices, not just phones, but also laptops. It's very common these days to have people, the workforce distributed, work in a hybrid environment. So these type of solutions are important. And talking about that type of communication, Silence Edge as well, which is a zero-trust type of remote access solution. And our SecuSmart suite of solutions as well for secure communications. Yeah, so secure communications, I wanted just to touch on that. Again, you know, I think a lot of the history, I guess, of BlackBerry was that the, you know, the handsets were these incredibly secure communication devices.