Digital Forensics with Emre Tinaztepe

0:00Digital forensics is the process of identifying, preserving, analyzing, analyzing, and presenting electronic data for investigative purposes. It's often related to addressing cybercrime and is crucial in tracing the origin of breaches, recovering lost data, and security hardening. Emery Tanez-Tepe is the founder and CEO of Binalyze, which is a cybersecurity company specializing in digital forensics and incident response solutions. He joins the podcast with Gregor Vand to talk about his path into engineering, his time in the infantry, Binalyze, digital forensics, and more. Gregor Vand is a security-focused technologist and is the founder and CTO of MailPass.

0:40Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at Vand.hk.

1:07Hi, Emery. Welcome to Software Engineering Daily. Hi, Gregor. It's a pleasure being here. Thank you for the invitation. Yeah, great to have you here, Emery. Yeah, we've, as we might get into, we've met once before, back when I was at a company, Black Panda. And as we might hear, that's now a customer of Binalize, which is what we're here to talk about today. So normally, I kick off these episodes with a background of yourself. I'm actually going to just add in one pre-question just to help, I think, our listener base because we're going to be talking a lot about digital forensics today. And I think just to make sure there's no misunderstanding around what we're talking about, actually, could we just define digital forensics and then we'll jump into your background.

1:50Sure. So digital forensics is actually a pretty old profession. It's an industry on its own. These days, I see that it's kind of seen as just a feature when it comes to endpoint security, but it's actually an industry on its own. And it's even longer than the endpoint security industry. So it's basically the art of collecting evidence, preserving it, and analyzing it. And the way we define digital forensics in traditional aspect is this. Collecting it, preserving it, analyzing it, and then presenting it to the court for solving an investigation. but what we do is a modern way of doing digital forensics which we'll be digging deeper into exactly exactly okay but that's great we've got a sort of a basis of what is digital forensics so i will now go to the normal start which is tell us a bit about yourself you know before founding vinylize love to just hear sort of how what was the road to founding vinylize sure before we start today i learned you were the one who refers our product to black panda and also our chief investigator now.

2:51So you guys were working there together. That's another indicator of developing a product that solves a challenge that addresses a need. So thank you once again for the invitation. That did happen, yes. Thank you so much. 39 years old, I started coding at the age of 11, 12. So it's been a very long time. And I don't remember myself without having access to a computer, not coding something. So maybe like two months, three months break, but I was always developing something because that's the passion. And the way I started was like, we had five computers at that time. And we were sitting, I clearly remember that moment.

3:30We were sitting like three students in front of every PC. So we were like coding interns. And I think this is very important because the teacher who taught us how to code, she said, this is not actually a part of the semester. So this is not a part of the curriculum. I'm supposed to teach you Windows 95, but I studied archaeology and I wasn't able to find a job. So I'm going to be teaching you QBASY because that's how I started making money. And that's how I decided to be a computer teacher, like programming teacher. So she said, I'm going to teach you just in case you use it in your career. So that's how things started for me.

4:09And then again, I'm an ex-Miltree. I started military high school after that one, after the secondary school. at the end of every computer class our teacher was giving us some like 10 minutes for us to like play or do whatever we want because if he didn't then people were finding ways to do it like in between the class so in the last like 10 minutes i was coding something on cuba's i was probably writing some like for loops like printing some numbers to the screen and then he asked me what are you doing i said i'm learning cuba's and at that time i was also like jumping to pearl which was another programming language, which was quite popular at that time.

4:46So I mean, these are happening at a very early age. So I'm really grateful for meeting these teachers. And then I started Military Academy in that period again, I had like access to computer all the time, and started learning Massen 32, which is Microsoft Assembler. So I was going deeper and deeper, started with QBasic, and then Perl. I remember working with Java for around a year. And then And PHP was getting quite popular those days. So we were designing, not with frameworks. So I really envy the ones who started later because there are no frameworks for that. So you were basically writing everything from scratch.

5:20So all the content of that web page was delivered by code. There was no middleware in between. So I started to go deeper and deeper. And I remember at the last year of the university, it was Muslim days, Microsoft Assembler, which was basically 32-bit instructions. set. And I was fascinated with it because that was the moment I understood that, okay, all these codes that I've been writing is basically translated into this that executes on the machine. So yeah, that's how it started. And my career has nothing to do with computer science, actually, because I was in infantry. My first mission was in Iraq.

5:59So I stayed there for two years. I was an infantry pro trooper. And in that process, I proposed 14 projects. None of them were accepted because, you know, military is quite strict when it comes to innovation. That's not the best place to do stuff if you're coming up with like new ideas. And my role had nothing to do with innovation. I was an infantry. So I proposed 14 projects. These were like mainly robotic like devices, like small circuits and some software projects as well. None of them were accepted. And on the 14th one, I decided, okay, I think I'm in the wrong place. So I shouldn't be here.

6:32I should be somewhere else in private sector. and I resigned and I was offered to work as a malware researcher because those days I was like digging deeper into how to reverse engineer malware, how computer viruses work. That was quite a fascinating idea for me. Like having a small, a few kilobytes of binary that can do stuff autonomously and then like spread around the world was, it also like sounded quite dangerous and I wanted to know how it worked. So I started as a malware researcher and then I received an offer from Komodo. I led their mobile malware research team for a year. And that was the moment I understood enterprise is not for me because I really missed being in a startup.

7:10My first career opportunity was in a startup. And then I returned back to that startup that I joined after the army, this time as a shareholder, and then worked around like seven, eight years. It was quite an experience. So I learned what to do, what not to do also, because startups are great for learning what not to do. And then I started Planalize. So that's the quick background on me. Awesome. So I mean, I think it's fair to say that, you know, a lot of founders, they end up founding something that's solving a problem they've personally experienced. So I guess sort of leading on from the sort of history, leading up to that point of what you've just explained, was there some kind of something you'd experienced already that then drove you to say, well, finalize, this should be a thing?

7:57Like what was the sort of moment there? Actually, my background is not digital forensics, but I was pulled to digital forensics because of like working with our advisors. So the first time I realized that there is a need was, I met one of our advisors and they were looking for someone who was like going to help them spot an insider case. So there was a malaria infection or claimed to be an malaria infection, but this needed to be approved. So the claim is, like the person, like the suspect was saying, I did not do it. My computer was hacked and it was done by the attacker. So we had to crew this.

8:33So it was a combination of forensics and reverse engineering malware analysis. So all of them were getting close to each other around 15 years ago. And then the second investigation, this time with our second advisor from the police department, they had a big financial institution, another breach. and the claims are really high. They will get this from the insurance provider. But the problem is the FTK, the platform, like the software they were using at that time, it shows a file, but no one knows what the contents of that file is. So it shows that potentially encrypted. And the interesting part was this file was starting as an outrun.

9:13So it was like automatically running when the machine rebooted. It didn't make sense. So, I mean, it shouldn't have some like encrypted contents if there is some data. And then it turned out to be a trick that was used by attacker to run another binary. So these were the indicators that traditional forensics was actually coming to evolve in a way that traditional antivirus industry evolved. Because when we started on the antivirus industry, first it was the on-demand scanning. So you were scanning your computer. And then antivirus companies introduced new methodology, like on-access scanning. So this way, whenever you or a process accesses a file, it was automatically scanned.

9:53And then we came up with the idea of why don't we make it faster and introduce on-execution scanning. So if something is not running on your machine, why am I supposed to touch that? Because when the first anti-race was introduced, computers were very small. The hard drives were very small. But the last time I remember having a hard drive in my hand, it was two terabytes. So if I'm supposed to scan all those files, it takes like hours. So that was the moment we decided, why don't we introduce our own execution scanning? So these type of innovations were being done on the AV industry. But forensics was an exception because we can dig deeper into this.

10:29Why? But I believe it's because it's very traditional. It has its roots from the law enforcement. It's a very strict profession that does not allow people to innovate, come up with new ideas, because at the end of the day, you are going to the courts. Or that's the assumption. which we're going to discuss further. So we started experiencing these type of problems. And I was trying to basically solve the problems we were facing ourselves with our advisors from NYPD. And to be honest, I was quite frustrated because we were waiting for FedEx to ship us hard drive images, which didn't make sense. So some, like from time to time, I was asking them, can I get access to that machine over TeamMaver?

11:06So give me the access. I want to like check some stuff and then you won't be needing the disk image at all. That's how we started. So the need was there. and in simple terms, I wanted to sleep more because I was waiting for evidence to be like sent. Yeah, I like that. So it's, you know, it's solving a pain point of sort of the profession that you are in. And I would say like a similar one that's popping up more and more now is I've seen a couple of startups of SREs, like Site Reliability Engineers. They're very keen to develop tools that will make their lives, like as you just called out, so that they can sleep more as well because they're getting fed up of their routines and they know that, you know, obviously AI, et cetera, can come help them out on that one.

11:46So I really liked that, you know, you had a lot of skin in the game in terms of what this was actually solving. So that's very interesting. And then, you know, if we look at sort of what the product is, you know, a big part of it is the fact it's all around automation and the fact that it's cloud-based. And I mean, you kind of touched on it there, you know, you were saying you were waiting on hard drives being FedExed to you, but what was it that really led you to believe like and you also touched on the fact that it's quite a sort of we would say like a state industry you know where as you called out law enforcement has driven most of the direction of it up until a certain point in time so yeah what sort of led you to believe that being able to move this in a direction of cloud and automation was even possible and like that you could be the one to actually be the disruptor here like what was the thinking behind that great question actually it's another shift that we observed at that time.

12:39So in my previous company, we signed a deal with one of the largest telco operators in the US. And we were expecting to have around like 50 ,000, 60 ,000 customers. It was an anti-malware product. So it was running on their machines, scanning their machines. But it turned out to be much bigger than we expected. I remember looking at the dashboard and i remember seeing like 100 000 and we thought okay that's gonna stop now so because they already deployed more than we expected and then it became 200 000 500 000 million and i remember at the end of that period deployment period i guess it was around like eight nine million end users but before it hit that threshold we have started to like run out of bandwidth in everything like all the infrastructure was based on virtual pcs vpcs on data centers so So that was the moment we started to work with shifts with our currently SVP engineering.

13:35We are working at Pinalize together now. So that was the moment we decided to, okay, we need to solve this problem because we cannot solve this with a data center that is located somewhere in Germany. And then we started digging deeper into how we can migrate our infrastructure to clouds. And I remember when I first logged in into Azure, it really felt like a huge data center waiting for me to run. So it was much, much more advanced than consoles of data centers. So everything was moving to cloud at that time, except the forensics, because forensics was supposed to be shipped with FedEx and DHL.

14:09So we were persuaded that if we want to make this technology available to enterprises, there is no way we can do this based on the traditional methodology. And the AV also, like antiviruses, also had a similar evolution. So first it was antivirus and then endpoint protection platform. And then they started introducing cloud antiviruses, which was, again, quite a paradigm shift because people were like against the binary going to cloud. But then comparing it against the tradeoff, I mean, do you want to be safe or do you want an executable like sent to some cloud? So people started to like accept the tradeoff there.

14:48So it was hard, to be honest. The first three, four years of finalized was really hard because everyone was asking the chain of custody, where do you save the data? And that's the reason we introduced our on-prem version, the same architecture, the same functionality. But if the enterprise is not ready for that mindset shift, then we were providing them with the on-prem version and it was running on their environment. But most of them, even the most mature enterprises, they started to ask for the cloud version themselves because they don't want to worry about deploying a product, maintaining it, running the infrastructure.

15:23They just want the value that that product provides. So it was quite a shift at that time. And now Gartner recognized this as a new category. And guess what? The first letter in this new category is cloud. So cloud investigation response automation. They call it CIRA. So it requires us a lot of time and resources to talk with the customers, tell them why it has to be scalable. It has to be running somewhere, either on our machines in the cloud or on their environments. But that was quite a mindset shift. Yeah, so I'd love to dig into this quite a lot. We've already sort of moved to the point of, okay, today, Vinalize runs on the cloud.

16:03and there's also been a mindset shift of sounds like a lot of companies by this point that this is the way it should be done and it's okay and so long as they trust like the technology behind it that's kind of the key thing there in terms of i believe of okay so before cloud the binalized proposition was the automation aspect to it i sort of believe and sort of what that could lead to in terms of time reduction for people like yourself who was in digital forensics generally So what was the kind of catalyst, the moment that you understood that what could be done in this sense? Like you thinking, okay, something that's taking me weeks even can actually come down to hours.

16:42Like what did that kind of look like in terms of a realization? Actually, the traditional digital forensics market was moving really slowly at that time. And it really felt like it's at the end of its evolution. And when we started, this is something we haven't covered, by the way. So when we started, our product was a dongle-based product. So it wasn't an enterprise product. We were just trying to solve the problem of collecting evidence and analyzing it from a single machine. And they're also used to, you know, like police officers are used to having a dongle. That's how they use all the traditional forensic products.

17:16So we started that way as well. But shipping dongles were really hard because we were getting orders from Australia, from the US, like from all around the world. And again, like we were waiting for like preparing the dongle packages and then shipping them. And we were receiving a lot of feedback. And just for our listeners, some might not even, just in terms of age groups, dongle here is like a USB plug-in adapter, right? You know, so if it's a physical thing, you plug in and it's effectively like a USB drive. It's kind of a USB drive that also has some licensing unit in it. So it's both for saving the evidence and also activating the license.

17:53And law enforcement is very familiar with this approach. Yeah, okay. And then the customers started to ask, actually. Customers started asking, I mean, we really like the product, but can we run this remotely? And when you talk about running it remotely, then there shouldn't be any dungle. So then we released the first version that does not depend on, it's called soft licensing. So we called it like dungle licensing and then soft licensing. And then the needs also started to come from the enterprises. They were asking us, can we integrate with our CM? and based on our previous experience, previous startup experience, listening to customer was a big part of developing your product.

18:28The way I describe it, even now, if you go to our release notes, you'll see every release has at least two, three credits given to our customers because customers are asking the releases, like the features that we release and we are giving them credits, release note credits. So basically the disruption was requested by the customers. So can we run this remotely? Can we integrate it with our CM? Can we run it with our SOAR? So these are the platforms that they were using at that time. EDR, XTR were not that popular at that time. So CM was the most popular product. So that persuaded us to see that there's disruption needed here.

19:03And also, I remember having NCASE certification books and checking the release notes of NCASE at that time. The product was basically not being updated anymore. So they were basically maintaining the product, not adding disruptive features. and also our advisors were like constantly asking can we integrate your anti-malware SDK with Forensic Explorer I remember that was one of the programs that they were using or like this type of like integration because we're constantly coming from the advisors which was also like persuading me right there's something needed here we need to like focus on this yeah I mean it's a good problem to have when it's not like you know for lack of people wanting to use the product or using the product it's actually they're saying we're using the product but would use it even more if you could provide it in this other form and i'll say that's kind of a nice thread that you were able to pull on and move along to not dongleize vinylize so maybe let's sort of jump into more from a technical standpoint just kind of actually what's going on here like what's happening kind of semi under the hood so am i right in saying that today by the product we're talking about is called air is that that's a vinylizer yeah ai is that kind of acronym for something or it's automated investigation and response right it was initially our product name but now there are three products that has the name air in it so it's kind of becoming a category name nice based on what i've been observing for the last few months okay so and this was actually a very interesting thing when i sort of first came into the space and you know admittedly i didn't need to handle a lot of this side of things i'm mainly talking about sort of my time in black Panda, you know, I didn't need to really handle any sort of forensic collection or incident response sort of directly, but I was made very aware of one of the big challenges there is the, you know, cross OS, you know, like across system architectures is it, you know, Linux, Windows, Mac, Android, you know, et cetera.

20:58So how does Air handle, you know, we're talking about automated collection. And I think that again, that was a, almost an argument put to me as to why this was so challenging and could not be automated was the cross os aspect so how does air actually handle that in terms of across the different operating systems so that was actually one of the selling points because we realized the fact that in order to be a good investigation platform we need to support multiple operating systems my background is also windows windows operating system windows external and at that time macros was getting quite popular in enterprise environments especially developers were using macros but it's going like much more common now and when we check the market reports macbooks were on the rise and then chromebook also like showed the same patterns and as far as i know they even like they were above macbook sales now so we were getting the signals i mean we are already like really good on windows now it's time to like focus our resources on macros linux and also chromebook so currently we have windows linux macros ibm ax chromebook and even ESXA.

22:06So this is one of the hard parts of digital forensics and combining this data in a single, like in a unified hub is also another challenge. So that was a need. And some of the customers were like specifically choosing our product because of the cross-platform support. Okay. I mean, there must have been challenges though, sort of in being able to handle the different, you know, so was it that it started with Windows, but I think you said moved to macOS fairly quickly or, I mean, what kind of things, especially sort of maybe just slightly higher level have to be considered when trying to create the same product in this space across the different operating systems i think the biggest thing is biggest challenges mac os side because mac os is a closed operating system so even when you on windows you can get a lot of like details if you want to dig deeper into the operating system architecture the kernel the forensic side of the things but when it comes to mac os it's not that easy.

23:03So even the books that are written on the subject are fairly old. So it requires us like a lot of research on the Mac OS side. And then once we had Mac OS, Linux was much easier for us because the architecturally... So we developed two products. The first one was for Windows specifically, and then we developed Tactical Cross, which was collecting evidence from both Mac OS and Linux. And we used choosing the language, choosing the framework, and then And making the research was the hardest part on that. Yeah. And I think, you know, just sort of pointing to recent cases like EG CrowdStrike. And, you know, why was that such a problem?

23:39Well, as you called out, you know, MacRess is this closed system where access to the kernel is not possible, basically. And that's where Windows differs. And, you know, I think it's often been misunderstood as to why, you know, people say you don't get viruses on Mac, you know, period. And it's sort of obviously not entirely true, but it's more along those lines where Windows was just built with that from that standpoint of the kernel can be accessed actually and Mac can't. But yeah, inherently makes when people want to do good things and actually be able to investigate, it obviously makes your job a bit harder when it comes to actually creating a tool that's supposed to be figuring this out for say macOS.

24:15Yeah, and macOS was not targeted. When I was in malware research, macOS was not targeted as much as Windows. And even Android was much higher when it comes to like modular numbers. So that was the reason. But currently, macOS forensics is a big need in the industry. Yeah, I think that's just a very interesting evolution there. In terms of, you know, forensic investigation, we're kind of talking about the difference of, you know, how quickly can we do this, but also then how thorough is that? How many files have we actually pulled up and what have we gone through? So how does, if we're automating that, how are you, you know, when designing the product, like how are you considering the speed versus the thoroughness?

24:54And is it a trade-off or would you say you've actually kind of figured out the win-win of being able to do it much faster and just as thorough? That's actually where the domain expertise gets into the picture. So if you try to collect everything from that machine, then you basically go back to traditional forensic days. and then there's no point of like designing a new product because it's basically like getting your full disk image so you need to find that balance that gives all the value without slowing down the investigation so that's how our product is able to deliver the investigation times from weeks to hours so there's a balance there i mean you cannot collect everything it should be incremental it should be like gradually like increasing you should be able to but you shouldn't in the first stage that's how i define it yeah okay makes sense especially you know in today's world what if any you know in terms of machine learning does that play a part in building up i mean this is not a area product that i'm like super familiar with from a pure technical standpoint so you can help us out here like this is a case where you are able to bring in machine learning and then sort of have these, I guess, like catalogs of understanding of what's bad, what's good, and et cetera.

26:10I mean, talk to us about that. Exactly. So that's actually one of the biggest differentiators because in traditional forensics, you have a desktop-based product that runs on a single image by an individual analyst. So like there is one-to-one-to-one relation in that investigation. But in our case, in modern approach, you can have access to thousands of machines in a single platform that gives you a baseline to like what is normal what is not normal when i see something on one of the machines is this also like the case in the other devices as well so you have access to this and based on this information you can easily use machine learning or even like ai now to get an understanding of like much faster understanding of what happened on that environment because we're not talking about a single machine anymore.

26:58It's an enterprise problem. So you need to have access to all the assets that may be involved in that case. And that's obviously sort of touching on the cloud component. And with, I think, when you were talking about sort of the concerns around clouds, you know, when you were thinking through the move or shifting the product to be able to run in cloud, basically, the big sort of pushback is around anything to do with data privacy and compliance. how do you handle that and you know you're vacuuming up effectively all the data off a machine and you know there has to be some pretty sensitive data there how do you work with that when it comes to cloud sure so basically the regulations and certifications as long as you have these certifications then you don't face that much of an issue but still there are always some questions but when it comes to which one is important so business continuity find the root costs, finding the root costs, or having our data in the cloud.

27:53And especially with the migration from traditional on-prem-based antiviruses to EDRs and now XDRs, almost all the XDRs on the market are cloud-based. So they already started that shift. In our early days as a startup, it was hard for us to train, increase the awareness, educate the potential customers. But with the XDR, the shift to XDR and CM products, this is already happening. So these are already cloud-based products and the customers started to ask like can we get a cloud version can we get a sauce version of your product but when they ask this you need to like provide them with these certifications and as long as they see that it's not that much of an issue now there are still some enterprises that especially government and military we also have like military and government customers some of them they still prefer to have on-prem environments but that's not that common now especially in the last year.

28:47Yeah. When we're talking about cloud, is the majority on your cloud, on private cloud, is a pure mix? How does that look? They mostly ask us to host it because they don't want to deal with maintaining a platform. It's already very hard for secure cooperation centers and MSSPs also. They're fairly compact teams. So we're not talking about hundreds of people on MSSPs because we have two types of customers, MSSPs and enterprises. And on enterprise, it's even a bigger problem because it's really hard to find people for the secret operation centers. And it's hard to retain them. So if you ask them to maintain a platform, then it becomes even a bigger problem for them.