Canary tokens and digital tripwires (Interview)

25:34Again, one of the things that canaries get lucky with, and I mean canaries as the overarching technology, is it doesn't matter how attackers get into your network. Like super stupid attack, they bribed someone and have their username. Super, super high tech, they were actually hidden in the firmware on your monitor and now they've broken out into your network. When they're on your network, they have objectives. Like they've got to go get something and touch something. And that's when they touch your canaries and canary tokens and reveal their presence. So it almost doesn't matter. And in a world where attacks become more prevalent, it almost matters more because it doesn't matter what the other noise is.

26:24what matters is someone tried to use this API key that was on the server, you've got a real problem. And so it becomes a really high quality signal for really low implementation cost, which is why the technology ends up working so well.

26:43Jerod Santo:Let's make a dividing line just so we got some clarity on this conversation. So you mentioned Canary. That's a hardware device, right? Right. That's something that you deploy. Yeah. How big is it? How big is it? Describe a canary. So they started off as. All right. Harun's going to grab one. I'm going to rip out really old ones. So when we started with version one. Audio listeners, you got to go to YouTube for this. Okay. This is Harun showing off a canary. So go to YouTube. Yeah. So this is janky version one. um and today we can you show me all the parts of it like the the interfaces the power like give me yeah sure do a product demo while you're talking if you don't mind show me all the parts so we make these uh so the hardware device that you're looking at is by design made to be dead simple so we used to say in our product demos that it should be like a kitchen appliance so that you can't do this wrong there's literally just this one button on it which is also its led And so the logic is you plug this in, it boots up, and it's going to show up as a Windows box on your network.

27:56Now, this device also talks into a console. So if I bring up a... On AWS, right, if I recall correctly. Yes, exactly. So it sits in AWS. So if I bring one up here and I'll share my desktop.

28:12Jerod Santo:Yeah, do a little screen share. We're getting a treat here, y 'all. Audio listeners, you're missing out today, okay? I'm sorry. If you're driving right now, pause the show so you can get to a screen. So what we're going to see here is just a standard console. Now, this is a home console that I've been messing about with, so you'll see a whole bunch of what we call flux, where because they're canaries, you can logically group them into different. Ah, very smart, yeah. And yeah, we like our BIRD acronyms and we use them far more than we should. And so what you're looking at here is on my home network, there's effectively two canaries.

28:56One that believes it's a Windows server and one that believes it's a Linux box. And so this Windows machine, if I click on it right now, you'll see it's running RDP, Windows Remoting, Windows File Share. And so what that means, so what I'm going to do here really quickly is grab its IP address. and that means that if an attacker on this network happened to try to browse to this machine, they'll see it on the network exactly like a normal Windows share and as soon as they browse to it, so when you create a canary, you can enroll it in Active Directory, you can RDP to it. So in this case, they're going to see a bunch of files and if they try to access one of those files, you're going to get this message telling you, listen, somebody just tried to connect to this thing.

29:58Here's what they did. Here's what they tried to see on this machine. And again, at its simplest, that's what Canary is made to be.

30:08Jerod Santo:So in a separate term, were you SSHed into that? You tried to hit it with something? Maybe you curled it or something like that? Exactly. And in this case, I just opened a file share to it. But the joy of Canary is, like, if you look at this guy who thought he was a Windows box, I can say, okay, I don't want you to be a Windows box. I want to reconfigure you. And so I can go in here and say, listen, I don't want you to be a Windows box. I want you to be a changelog demo, if I could type, NAS. and where we get its personality instead of a Windows 8 machine, I go in and here's all the options for what we can make these machines, these canaries just at the...

30:58Ooh, make it a TrueNAS box.

30:59Jerod Santo:Can you make it a TrueNAS box? So I think we've got in terms of NASs, if that's the plural of NASs. Let's get TrueNAS on a list if it's not on a list. Let's see who we've got down here that we can use. Maybe even a ZFS file system, whatever works. Yeah, exactly. Yeah, Spelunk is fine. Sure. We don't use Spelunk, but sure, we'll go there. Yeah, so let's make it a Synology. Synology, yes. Right? And you'll notice when we do this, it changes a whole bunch of defaults. So it says your MAC address is now going to be a Synology. It's going to change, like, is PortScan detection enabled? And I can just leave all of that.

31:39So you'll see it says this is what the web server on it's going to look like. So what I'm going to do initially is just leave all of the defaults, except I can go in and say, look, I want a file system on the Synology NAS and make my file system specific to my industry, my cybersecurity industry. And so it's going to say, okay, I'll create a file system for you. And we can even go in there and say, look, I want to create a new folder called change whatever. My spelling is bad. Demo. And in there, I want to create a file, a password protected doc file called secret.doc. Okay, so you can leave the defaults or you can do this, and then you say deploy.

32:34And literally what's going to happen right now is that canary, so I'm going to see if I can change what you're looking at on the camera just for a second. If I change to this, you'll actually see you can turn off or you can't switch cameras. Yeah, they may get upset.

33:02Jerod Santo:They may get upset. This is not that sophisticated. Riverside, I hope they didn't mess up. So I thought I'd show you the actual canaries, but that's cool. What's going to happen now is that canary basically gets told to reboot. And when it comes up again, it basically stops being a Windows box. It comes up. Initially, the hardware light goes red, saying, I can talk to the console, but I'm not encrypted yet. And then it goes green to say, OK, I'm now on the network. and what you'll see as soon as this guy goes green again, which he now will, is you'll effectively have a disk station NAS on your network.

33:49And so as a Windows box, you could RDP to it, you could enroll it in Active Directory, you'd have Windows RPC. As a disk station NAS, it'll now have disk station NAS services. If you made it an IBM mainframe, we do a lot of work. So even down to the network level, like you TCP, you nmap it, the stack fingerprint looks like it's supposed to. You make it a Cisco router and that's what it's going to look like. And so for a defender, really the amount of work should be minimal. But if they want to do more complex stuff on it, they can. So I've just hit it. So you'll see this now says that we've got a changelog demo NAS here.

34:40And if I click on it and grab its IP address, if I open a new tab, is that still going to be shared?

34:47Jerod Santo:I don't see the new tab, no. It's okay, though. We'll follow you. So let's stop sharing. You can probably stop sharing and then reshare a new tab. Yeah. Got some post snafu or some post food on here. Go to that. And so if I surf to that guy, effectively, it's going to look exactly like a Synology NAS should. And again, an attacker who's on your network and now sees this, like they at least have to try an admin admin, right? And what you're waiting for is for that one alert that tells you, listen, somebody just found this thing and somebody just clicked on it. And here's the creds that they tried.

35:40And this starts to get deeper, right? Because if they tried valid creds, they tried Bob from Accounting's creds, you now know that Bob from Accounting is compromised. And so, again, it's at such a fundamental level, how can attacker resist this?

35:57Jerod Santo:Yes. For you, it's so easy to deploy. And so those were canaries that could be like that. And if you say you want to add a canary, so I showed you the hardware version, but you could just as easily say, look, I want to add a canary, but I want it to be a VMware canary. or I want it to run in my GCP cloud. I want it to run in my Oracle cloud. And the exact same thing happens. A VM pops up in your network and when an attacker touches it, you get this alert when it matters. So that's Canary. And then within Canary, each customer gets these Canary tokens, which are exactly what you've seen on the publiccanarytokens.org, except you can go here and say, I want to create an AWS API key.

36:53And I say, I want to put this on Adam's laptop. And I say create. And what this gives me now is, okay, here's an AWS API key. Put this on Adam's laptop. We put it on, we forget about it. and again you can forget about it forever and when an attacker who you care about and and that's the whole joy of these like whether you've got these uh canaries on your network or canary tokens deployed somewhere put them forget about them and when an attacker that you care about finds them you're gonna know and and look some of the tokens are just cute so so i spoke about the credit card token, you saw the DNS token.

37:45But for example, like if you use WireGuard, like we'll give you a WireGuard endpoint, store it on your boss's phone. And when he goes traveling the next time, and you worried that he's in some foreign country, and they've sucked down all his VPN information, you're going to get an alert saying, look, the VPN that was only on your boss's phone just got accessed. Or we'll give you a QR code token and say, we're doing this for Adam's test. And when it's over, redirect to google.com. Redirect to HTTPS, google.com. And so here's a QR code. And the funny thing about this, again, like I think last week we tweeted, somebody hit us up to say, listen, they did a user awareness training thing on their network.

38:47They printed a few of these QR codes, left them lying around. And after everyone went home, the QR code got hit. And it turns out that whoever was taking out the trash was going through the trash and effectively dumpster diving. And so literally the tweet on the page is like, they didn't know they had this problem, but it turns out that the people going through their trash are doing stuff with it. And so again, it's such a nice case study for canaries and canary tokens that say, even if you think you don't need them, drop a few because it's going to take you five minutes and cost you almost nothing.

39:35And if it surprises you, it'll change your plans. So you should consider it. So that was back to your question of the difference between Canaries and Canary Tokens. Canaries are entire operating systems that either run on hardware or clouds or VMs. And Canary Tokens are smaller tripwires that you get to put everywhere. and canarytokens.org are completely free. Anyone in the world can go hit it, enter your email address, not so that we can mail you, like we're not going to mail you or try to sell anything to you ever, but literally it's so that when someone trips that tripwire, that's where we'll alert you to say that badness has happened.

40:26So you can start just by hitting canarytokens.org Create a canary token or 10 Get alerted when it matters

40:40Jerod Santo:Super cool, super cool That's super cool, man Thanks, it's fun

40:50Jerod Santo:Well friends, I'm here with the CTO of Buildkite and one of the most challenging problems of modern era software development is continuous integration and continuous delivery. And so Lachlan Donald, BuildKite CTO, what are you thinking about today's teams, the challenges they face, the speeds at which they're developing new features, new code? It is just overwhelming. How do you all think about that? Such a good question. It's the question everyone's asking right now. All of our big customers are asking us at the minute, Like, you know, if we 5 or 10x our throughput this year or 1 ,000x it, what breaks and when?

41:28And, you know, my answer is kind of same as it's been for the past 20 years, which is that the bottleneck is still trying to integrate those code changes in and then deploy them and check they work and then keep them working as you keep throwing more and more code at it. I think a lot of the fundamentals are the same, but we're just 1 ,000xing the speed of it. And, you know, that changes nearly every variable.

41:51Jerod Santo:Yeah, for sure. Okay, so where does BuildKite thrive? What particular type of team or enterprise do you thrive in? The area that BuildKite has always thrived in is like this like fastest moving tech companies of the world. Like we've been disproportionately successful in that small niche. The kind of Shopify class, Uber class, you know, OpenAI class of folks that have this key problem around iterating really, really fast. And, you know, the thing about all of those folks is they all have subtly different needs, subtly different problems. And so we've tended historically towards building like really well engineered Lego blocks that scale like orders of magnitude more than what our nearest competitor does.

42:37So, you know, I think that that puts our system in this tension where, you know, you've got to spend some time assembling those building blocks, those Lego blocks to get the thing that you want. But the end result is far and away more performant and scalable. And the experience is better than what you get from something that's off the shelf. So I think we've started from a position of really well engineered logo blocks and then are kind of working backwards towards kind of creating the thing that scales down to a startup that starts with one person and 10 agents next week.

43:09Jerod Santo:Well, friends, go to buildkite.com. That's buildkite, K-I-T-E dot com. You deserve better CI. Engineer for the frontier we are all facing, trusted by the teams setting the pace. Again, buildkite.com Once again, buildkite.com

43:35Jerod Santo:Yeah, I can't help but to say wow Well, there's a lot of wows in there Wow at the technology Wow at your I guess just your desire to share with the world And improve it And leave an impact And not so much Not pull back all the money you can nothing that I want to sort of like focus on that but just wow from a lot of different perspectives so we got to that deep dive thank you for that demo that's phenomenal so one the device

44:06Jerod Santo:is small simple etc that was a v1 I think you said we were looking at maybe there's a v2 that's been improved but maybe it's got just one less button who the heck knows maybe I don't know how to improve something that's already so simple. When you host canaries, do you need one? Can you have one canary in a network? Do you have to have one for every time you're creating a canary? Or can one have like multiple, like many canaries on it? Great question. So we, by design, make them so that they can only ever be one other system. And like lots of us in the company have pen test, offensive security backgrounds.

44:48And we used to love, as attackers, finding anything multi-host with more than one network card on it, because inevitably people use them to straddle VLANs, and you can attack it on one network and pop out on another. And so, in fact, it's a good segue for something else. On our canary.tools website, we've got a whole page dedicated to security, but it's kind of unusual. So if you go canady.tools forward slash security, we've got a bunch of how we think about the security of our devices. And it's something that we think more security vendors should do. Because anytime you run someone's device or someone's software, you exposed like they could be the weakest link in your network now.

45:44And so there's a bunch of stuff that we do to make sure that canaries won't be the thing that ruins your day. And some of them, again, we've done kind of unusually. A blog post we put out a few years back that is a little surprising. Like we've got these canary devices and they report in to your AWS console, but we don't multi-host that console. So we don't have one fat web app that all our customers connect to. Even though that's easier to manage and is pretty standard. But from a security point of view, if we did that, at some point we'd make a mistake and some attacker would be able to log into their account and see your account.

46:31And so we don't do that at all. Like every customer gets their own hosted Canary Token server, and it gives us a sort of customer isolation that says no attack's going to bleed over from one to go to another. And it's really old school security isolation. But again, it's because we don't want to be the thing that gets our customers compromised. So, sorry, back to your question. A single device acts like a single operating system at a time. And what we do for that is make them effectively cheap enough. So customers who buy canaries effectively pay$1 ,000 per canary per year. And so when they start off, they pay$7 ,500.

47:20They get five of those devices. and then as they add canaries, they add them for an extra$1 ,000 per year. And that's just always how we've done it. And typically what customers do is they start off with$5 ,000, so they pay us$7 ,500, and inevitably we catch their pen testers or we catch an attacker, and then they say, hold on, we should put these at all our remote sites. And so, again, something that we were super fortunate for I mentioned we don't do outbound sales. We've got single customers now who pay us hundreds of thousands of dollars for seven years running. And we've never met them. We don't have a sales team to talk to them in the way that they normally do.

48:11We've got single customers now who pay us north of a million dollars. And we don't have the people to do that sale at all. But what they did is they started off with five canaries. It didn't suck. They moved on to 20 canaries and they liked it. And now they have thousands of canaries because it works. And so our pitch as a company is if we can keep making it good enough so that people renew and keep doing the right things, then we can keep focusing on that and not have to focus on shiny ads at airports or things like that. And so far, it works.

48:59Jerod Santo:Yeah. You know, I have some thoughts, I suppose, on all that. I think the shiny ads at airports, sometimes I was traveling recently, I think maybe in the last year. I remember seeing an ad for Notion and Notion is one of our sponsors. We use Notion. You probably use Notion. And the ad was just like, so not, it was like brand awareness. Right. I suppose. And I'll just dovetail off that for a second, but sometimes those shiny airport ads are not because they have to. It's because it's a cool factor. It's like, you know what I mean? Like it's like the Superbowl ad. We don't really need to advertise Pepsi or Coke one more time.

49:47Jerod Santo:Everybody in the world knows Coke and Pepsi. It's for the top of mind, mind share, brand awareness. Just because you can, you do. I'm with you. So now I'm going to ask you a culturally tough question. Okay, do it. Have you discovered paddle yet? Paddle? Yes. I don't believe so. So it's like pickleball that Americans love, but better. Okay. Americans love pickleball, yes. And here in the great state of Texas slash Austin, Texas, we actually have the pickleball capital of the world, or at least it was for a while there. It was called Dreamland right down the street. I'm sorry. It's not going to last.

50:33Paddle's going to take over. Paddle's taking over. So Pedal's this thing that started in Mexico and then went big in Spain. So currently Spain and Argentina are the world leaders in it. But legit, it's like a worldwide phenomenon. And USA is just starting.

50:53Jerod Santo:Or football. Yeah. And USA, like Miami has just started with it. But we discovered it as a company about three years ago. And we are crazy fanatical about it. Like there's a whole bunch of us who play a lot. And so this year, the current world number one contender has a thinks canary on his sleeve. And again, like you say, it's almost just because we can and because we think it's cool. And it's, yeah, it's totally just like for us, one of those things that go, we can and we think it's cool. And we love the sportsman. Like, other than being great at what he does, like, he's done some really cool things in terms of being a good ambassador for the sport.

51:43And we're like, yeah, we like that dude. Like, we think that dude is very us. And so we now sponsor the world number two paddle player. But, no, I'm totally with you.

51:57Jerod Santo:I'll wear a Thinkst Canary t-shirt every episode, man. I'll send you one. Okay? I'll send you one for sure. No, it's – in fact, a while back we did a post on – I don't know if we discussed it the last time – on the stuff that we give out as Canary gifts. In part because we are crazy particular about it. So throughout the year, we spend a lot of time looking for like, hey, we think this is cool and we really like, like we wear all of our stuff But other than us, we think that a customer or like someone who likes our stuff, who wears our t-shirt, like we're genuinely proud of it And so like we never do like cheap throwaway t-shirts that'll go bad in a wash These are nice t-shirts Our hoodies are really good.

52:56Like people really like them. Oh, I'll tell you something else that's super crazy. Like every year we do something else that's weird, which is we try to, at the end of every year, send all our customers a customer gift. And so this year we sent everyone these iFixit driver kits.

53:16Jerod Santo:Nice. Yes, I love that. And again, people find it. Where's mine at? People, well, if you're a child of the 90s, like a year or two ago. Swiss Army knife? Like, yeah. And again, it's the sort of thing you can, like we do because we can. And it's just the thing that says to customers, hey, like we think it's cool that you still support us. Thoughtful advertising in a way. Thoughtful placement. It's the sort of stuff that, again, we're not hippies. We like money, but we make enough money that we can. And the stuff just doesn't have to be junk. We genuinely appreciate our customers. And if we're going to give them something, it should be something that we like.

54:06And yeah, so far that seems to work nicely for us.

54:12Jerod Santo:I like that. I like all the way. That's why I said we got to get room back on the pod. Even if we go back over some stuff again, that's what I told you in the pre-call. I told Imadi and stuff like, hey, listen, you know, I don't care if we repeat ourselves from the last show or not, because I enjoy the way you think, Karuna. I really do. I think the way you think is a breath of fresh air. I really do. And I like the demo, too. I think that's so you answered the question, which was, you know, one, how does it work? And then two, you gave us kind of a visual demo. So audio audience, go to YouTube for this one here.

54:47Jerod Santo:And then you gave us a visual, which is cool. as well. I like the fact that the individual hardware canary is only one thing. That does make sense from a security perspective because you want to limit your exposure. And hey, you may have multiple. Maybe you have a different problem, which is, hey, Haroon, we have literally 150 of these things in our hardware rack. Maybe there's a 3D printing ThinkVerset thing out there. I don't I'm imagining I'm a racker, right? I'm a mini rack guy or a full rack kind of guy. Next thing I know, I'm going to get, if I got more than five or even one, I'm going to want to make it nice in my rack.

55:29No, so we've got both those. So we've got rack mount versions that customers get. And we've got customers who've just got a handful of them that have created their own STL files and shared them so that you could rack mount like four or five of them. Yeah, it's pretty decent. and you can get rack mount versions of them too.

55:50Jerod Santo:You know, not that I... I'm thinking about... Now I'm thinking about it from a tech stack. So I want to go back to self-hosting. I want to go back to Canary Tokens, the fact that I can self-host this Canary Token server and then juxtapose that against the fact that you have a hardware device, which is called a Canary. So they're not the same, but they're similar in the worlds. And from within a Canary, console that runs in AWS that's my own. It's not the SaaS version with multiple change logs or multiple customers on it. It's my own hosted version of it on AWS. Inside there, I can manage all the hardware canaries I have, but then I can also create canary tokens.

56:37Jerod Santo:Who's hosting those canary tokens? Where's that server living at? It's on another IP on AWS for you. So it's hosting the cloud. Yeah, exactly. So hosting the cloud for you. Actually, I'm going to throw in one other confusion in there because there's something else that we build and give away, which is we also do something called OpenCanary, where OpenCanary is the free software only version of these guys. And that's where I was going. I'm glad you're answering this question. Yeah, so it's super scaled down compared to, like with Canary tokens, it's absolutely as good as it gets. Like even our paying customers get Canary tokens that are effectively the same.

57:25OpenCanary is a limited core of this Canary device. So if you didn't want to pay anything, you can go to opencanary.org and you'll effectively get a tiny working honeypot that you can run. You can do a little bit of configuration. And again, we've got great users. So there's YouTube videos from people on how you can take OpenCanary and deploy it on simple hardware, deploy it on a droplet, deploy it on a Raspberry Pi. In fact, we've bumped into some customer networks where we've competed with OpenCanary. Like we get to a customer and the customer's like, look, we've got OpenCanary deployed all over.

58:15Why should we pay for you? and sometimes I'll think to them is, look, if you're happy running that, like we build and maintain that exactly for people who don't want to pay for Canary. So it might be that that's perfect for you and we're happy if you run that and that's cool. So Open Canary is the free little brother of Canary. Canary is slick and drop it and forget about it and it's running in two minutes. And Canary Tokens are free for everyone on canarytokens.org. And you've got your self-hosted option where you can pull the Docker image, you can run it from GitHub. And look, that has benefits on its own because if you take our canarytokens.org canarytokens, some of them will tie back to canarytokens.org.

59:14But if you self-hosted this, and you run it on adamnetwork.com, now when someone finds that Canary token, it links back to adamnetwork.com and it makes that token even more believable. So again, we make it easy for people to do that. And if they want to, they totally can and should.

59:38Jerod Santo:Yeah. For me, I think self-hosting, and especially in this era, like this is 2026, as we all know if you're listening to this. Maybe you're listening to the future like, gosh, these idiots were talking about that in 2026. So this is March 2026, and we're in the burgeoning era where the entire operating system of software development is literally being changed as we speak. It's being rebuilt on top of agentic AI. Agents are everywhere, and we're all learning right now how to use them. So if you're listening to the future, that's where we're at right now, okay? Future, I don't know where you're at, but it's probably cool because I know where we're at now, and the horizon looks pretty cool.

1:00:16Jerod Santo:And the reason why I'm so focused on self-hosting isn't the free nature of it. And I guess it is in the sense of freedom, but not necessarily the cost that I'm trying to focus on. It's the sovereignty. Is that I feel like my gut in this era is saying, even in the enterprise world, because the ability to create bespoke software that solves my need, SaaS is changing. They're talking about SaaS killing in a way there's a lot of things and i hate to use that word killing because it's just too negative um but there's a lot of change happening right now and i think for me specifically i'm focused on what can i build just for the fun of it myself what can and should i self-host myself not because i don't want to pay somebody because i'm cheap but because i want sovereignty i want control and a boundary that i can reason about that i can manage responsibility Not just the, you know, the responsibility to make it, but then the just knowing where it is.

1:01:18Jerod Santo:And I think Canary Tokens is this example. So we kind of got to the demo from that lens, which is, okay, wow, I can self-host Canary Tokens. And that's super cool that you have OpenCanary. Now let's talk about from a developer standpoint. Take me into this world. Okay, so you've got Canary, hardware device, running software. You've got OpenCanary. Are those two codebases divergent? Is it open core? Is there a build on? Help me understand how the codebases all interact. So those two are pretty different codebases completely. So they're not even the same core. They diverged so early on and yeah, they effectively two completely different products.

1:02:06Jerod Santo:How do you manage that difference then? I guess maybe not so much answer the question, but more like from a developer standpoint, because if they're completely different, how can they open canaries? Kind of a misnomer in a way, almost, because it's not really open canary. It's a fake open canary, because it's not the true canary is what I mean by that. So mainly, I guess it's if you think about what Canary is, right? So when we've got this thing that's going to act as a fake operating system, it's got to have a bunch of components, right? So it's going to have effectively a core that says run fake services.

1:02:49And so what we've done is we've said, okay, this service would be good and useful for people. let's rip it out and put it into OpenCanary. Like this is worth it for people. And then the components of it, like let's fake out operating system TCP IP stacks that won't get ported across to OpenCanary. Or what we've done a lot of work on with Canary proper, I mentioned in our original call, but when you drop a Canary device on your network, right? it's going to talk to the console running in AWS. And we wanted, like our promise is this stuff is so easy and it just works. And one of the things we do to make that happen is all communication between your canary and the AWS console run over encrypted DNS.

1:03:47And so on your network, you don't have to make holes or allow management traffic. point it to your DNS server internally. As long as it can resolve DNS, it bundles all of its communication with the console via DNS. It goes through. Even when we push updates to the device, which we do multiple times a year, the only communication between AWS and your device is DNS traffic. And that's not port 53 from your Canary to the internet. It's your Canary talking DNS to your internal DNS servers. And so that entire encrypted DNS channel only exists for Canary. Like it won't be, it won't exist on OpenCanary.

1:04:44OpenCanary is much simpler. it says, hey, point me to a mail server so I can send your alerts out via email. And if you take the Kinetic console, which we blitzed through earlier, And we've put a lot of work into making sure that that console isn't the center of your universe. So on day one, you go in and say, send all my alerts to Slack, and then never log into your Canary console again. Or, hey, I'm running Splunk, send all the alerts to Splunk, and just never look at us again. So Canary is really a super convenient version of Open Canary. But those two are sufficiently divergent. And like, I don't see that changing just because at this point, it would be incredible amounts of work.

1:05:40So OpenCanary is, hey, if you like the honeypot idea, you want a bare bones honeypot without a lot of work, use OpenCanary. Canary tokens is like, as far as we're concerned, our cutting edge stuff on tokens and you get it free and self-hosted. And Canary is the thing that pays our bills.

1:06:10Jerod Santo:Nice. Yeah, I think even last time I was Alluding back to the fact earlier I said I've gotten smart since then I think I said I wanted like, hey, I self-host My pile, I want to self-host I want a canary, I think I even asked you on that podcast I think you said you'd send me one, but you never did But that's okay I'm just messing with you No, no I don't even have an AWS account So if I have to run my console In AWS, I would have to go and become an AWS user to get one. We didn't even do it for you. So we make that stuff so slick. I'll drop you an email after this. It'll be interesting to see your comments playing with it.

1:06:50I want to play with it.

1:06:51Jerod Santo:I mean, it's less like I really don't. I don't really need it. I just want to play with it. Like I'm in this era where, especially as a podcaster, I mean, this is the benefit of the job, right? I get to play with everything. Yeah, it's a win. And I would love to play with it. We'll make it happen. It's totally for you. I'll even do a demo. I'll do a demo video of it. Well, I'm starting to do more stuff on our YouTube channel. I'm starting to break into that a bit more. And I think this is an area where it would be cool to show off this kind of device. I mean, I think, again, back to the fact that you've got this, not a pejorative, but this tiny little company that does, was it 20 million annual recurring revenue?

1:07:30Jerod Santo:Is that accurate, Arun? Yeah, so we've now gone a little past that. But yeah, so last year we crossed that. 22 and a half ARR? I'm just kidding. 30? Yeah, no, it's pretty cool. And again, we've never raised our prices. We're not doing outbound sales. Like we don't ping people and say, hey, you should take more Canaries. Like we think if we can do it and not suck, then people will use us more. And again, like one of the caveats that I want to add is we don't lack for ambition. Like I want, I genuinely think Canaries should be on every network. Like if you're a big company, you should have a few.

1:08:16And if you're a small company, you should have a few because like they save people from genuine badness. But we just don't think we have to blast it to the world. Like we think the way we're growing, we'll get there. And, yeah, that's just how we plan to take over the world. One happy customer at a time.

1:08:38Jerod Santo:Let's go back to, if you don't mind, since this is GPL, let's go back to Canary Tokens. So github.com slash thinkst slash canary tokens. No, just canary tokens.org. No, no, I'm actually pointing out the GitHub repo. Oh, the GitHub? Oh, yeah. Yeah, sorry. I was trying to direct the audience there. So that's why I'm camped out right there because the reason why I ask this question is, one, open to contributions. It's open source, but do you take any contributions? Yeah, we do. In fact, late last year, we bought a small UK company that was building Canadian tokens. And we like what the engineer was building.

1:09:21he had sent us some stuff but he was actually building his own Canary tokens and we spoke to him and said hey, we think this stuff is cool like why do you have to have the overhead of also building a company like come build stuff with us and so it was our first company acquisition but yeah, other people who want to build Canary tokens and people submit patches in the open source tradition, like it's not a ton of people who submit, like clearly more users than submitters, but we'll happily accept PRs. And yeah, it's a good idea for people to check it out and play. In fact, almost all of the tokens that are significant, when we release them, we also release a blog post that often says, here's how we built it.

1:10:19So like the AWS Canary Token, we would have gone into great lengths saying, here's what it actually does. Here's how we're getting these logs out from AWS. Here's the trick behind it. And in some cases, what that means is some corporate network somewhere might decide to not use our whole Canary token framework, but they can use that trick to set up traps on their network and more power to them. Like one of the benefits for using Canary and Canary tokens is lots of times you're a big enterprise and you think, well, I can build this honeypot. I can build this strip wire. And then priorities in the company change and someone rolls out of the company and who maintains it?

1:11:08And who maintains the alerting? Like, did the alerting go down and now you've missed your critical alert? And so people pay us when they want that to not be their problem. So if you are going to do it on your own, like, we'll tell you how we did it and we'll release code and you can go do it. But if you want to do your business, then we take care of this business and it's a good deal for you. And that trade-off seems to work.

1:11:39Jerod Santo:I'm going to take this back to my lab, Harun, because that's why I'm asking these questions. Because one, when I start touching a code base, I can't help but just find, I'm an idea guy. You can probably tell that. And I can't help but start to solve my own problems. And that means potentially changing your code base. I'd love to be able to understand the pattern of what it takes to create new canary tokens. Because the one thing I, and maybe, maybe there's a reason for this, but when I look at canarytokens.org slash nest, what I don't see here is an SSH key. Yes. And that seems like a pretty obvious one.

1:12:17So it's interesting. So, so if you did an SSH key, you'd need an SSHD to catch it. Right. And, and so we'd have to host an SSHD and we actually. It's so interesting. Go into the nerdy stuff, man. Take me there. What's the angst? No, so what's interesting is I skimmed past it when I had the console up, and you actually take me straight back into it. So that's a great thing.

1:12:46Jerod Santo:Go ahead. Console's coming back up. Let's see it. So we've just released what we call, and it's going to be slightly confusing for your audience because I'm now going to introduce a third concept. And it almost doesn't have to be a third concept, but this is what we have. But literally, you're looking at something that we've just announced this week. So if we go back into the console, you notice the canaries that I spoke about and the canary tokens that we've spoken about. And down here, you notice something called breadcrumbs. Oh. And what breadcrumbs are, breadcrumbs are like tripwires, but they lead you to canaries.

1:13:37So like Hansel and Gretel's breadcrumbs. And so if you take a look at these breadcrumbs, it's a concept that we're just starting to make a first class citizen. And again, in the way we work, if you've got Canary as a customer, you just magically get the stuff free. But you'll notice the first one is exactly what you said.

1:14:01Jerod Santo:Yes, thank you very much. I am running this Canary on my network, which happens to be running a fake SSH server. Then I can create a breadcrumb to that SSH server. And so if I go SSH here, it says, well, which canary would you like to lead a path to? And I can say, well, I want to lead a path. The demo, yeah, the demo we made earlier. Yeah, so would that have been running SSH? Yeah, sure. And then I say this reminder is going to be on Harun's MacBook. And so what it's going to do now is create a set of SSH keys and give me a SSH config file that does two things because it points to that canary.

1:14:59Now I can take this and download this breadcrumb and store it on Harun's MacBook the way I just said. And an attacker who compromises me now sees these SSH keys that point them to a canary. And so it's going to increase the chances that this attacker is going to hit the canary on my network. And when they do, that canary is going to be able to say that key was created only for Harun's MacBook. And so you're now getting a double time. But if you're a large org and you just downloaded, you just bought five canaries, so you paid us seven and a half K, you can use this to create SSH keys for every server on your network.

1:15:52because if an attacker compromises that server, they're going to find that SSH key. It's going to lead them to your actual canaries and it's going to tell us which server was compromised. So again, it's just massively scales out the benefit to customers and it's free. So yeah, this could have been called another canary token because the concept is exactly the same. We see it here as a breadcrumb, but the concept is exactly the same as a canary token would be. So, yeah, it's a deployment artifact almost, but there you go.

1:16:38Jerod Santo:Yeah, I am a little upset with one more concept. Yeah, it's totally... Yeah, we try hard not to introduce more concepts because, again, the cognitive overload, like you end up with, oh, but do I do this or do I do that? And so we've had literally what you're looking at, like we pushed it out and announced it this week after playing with it for a really long time. and we've had customers trialing it and using it in anger for a while. And it's something we're going to do more work on going forward. Like we think it's worth it. And again, something that... Breadcrumbs, though, is that in the naming sphere of...

1:17:27Jerod Santo:I mean, that's where you sort of dovetailed off. I mean, I'm using a little pun there for a reason, But it doesn't fit the narrative of canary tokens. Yeah, it's true. I suppose breadcrumbs, maybe canaries follow breadcrumbs, maybe. A crumb trail? Yeah. We went to Little Grimm Brothers. Well, you might be in the park. But do you have canaries in the park? No, you probably have, what are those in the park? Like, what kind of birds are those in the park? Just doves? I don't know. Pigeons? Pigeons, yeah, pigeons. Those are the most popular in cities is pigeons. Okay, I'll follow you there. I'll let you have it.

1:18:08Jerod Santo:So the reason why I asked you that question, and I'm really not angry, but I, as a developer, I angst over this and everyone listening to this will totally understand this. When you start to build out a world of a solution and you start to name it, and then obviously you want to kind of keep that name. Yeah. I try my best to not fracture that world, but to kind of build incremental good layers on it. Totally. And I empathize with the struggle because, I mean, I pointed out SSH keys for a reason because that seemed like the obvious canary token. And I saw it missing from the list, but it's more of a breadcrumb.

1:18:46Jerod Santo:Why did you choose breadcrumb over token for that particular one? So interestingly, and maybe we will bring a token out that is an SSH token also, because again, the company that we bought in the UK actually had an SSH token. So we bought them and we bought their tokens. And so we now have a working SSH token pre-built in there. we've liked the thought for a while of leading people to canaries on corporate networks I like the idea too that idea of crumbs that says if the person has SSH like create SSH config files you can put one in the docs public docs can have little breadcrumbs exactly so if you just look at our initial list in an API you can be doing API sniffing and your API sends something unique that no one cares about because you're just trying to build on the application.

1:19:49Jerod Santo:But a hacker with a nefarious means will look at the API call instead and the header and say, hey, there's extra bits in there. What is that? And that's a breadcrumb. Exactly right. And in fact, when you just play with Canary tokens, you're going to see even the simplest of Canary tokens without having to build it or build stuff, you can build on. So you mentioned the DNS token, right? If you took a DNS token and put it in your host file and you called it supersecretserver.adam.home, then an attacker who finds it goes, what is this? He resolves it or he surfs to it. And the moment he does, his machine resolves that DNS token.

1:20:42And you get told, listen, this thing that was only stored in Adam's host file just got resolved. And if you take simplest, like one of our simplest, simplest canary tokens is we just give you a unique URL. And if anyone surfs to this unique URL, you're going to find out. We're going to tell you someone surfed to this URL. Now, years ago, a really smart reverse engineer gave a presentation on how you could embed canary tokens inside of binaries so that when people were reverse engineering your binary, you'd basically, so you could take one and make it a URL endpoint inside your code. So now the guy reverses your code.

1:21:31He sees this URL in your code. He goes, well, what is that? He hits that endpoint and you know somebody just reversed your code. And he actually spoke about different versions of it because if you wanted, you could wrap one up in a complex loop in your code. And now only the type of reverse engineer who unfurled that loop could get that URL. And now you know that you're being hunted by a slightly more advanced attacker. So the Canary tokens really give you detection primitives that you can use in different places to do cool things. And with them, really, the sky's the limit. Like you can genuinely do cool things.

1:22:18Yeah, just based on what you want to do.

1:22:25Jerod Santo:My friends, I'm back with a good friend of mine, Michael Greenwich. Michael, I know that I love WorkOS. Our audience may not know about WorkOS, but what are the challenges developers face starting a new project? Choosing the right tools, choosing the right database, choosing the right auth. Take me there. When a developer starts a new project, the decisions that they make at the very beginning end up having long lasting consequences. What language you build in, what platform you build on top of, what database you choose. These are things that are very hard to change later on. So they have like major consequences.

1:22:56And especially if they limit your ability to grow and scale, at some point, as the product starts to take off, you're going to have to stop developing new product features and go re-architect or rebuild your system. And that might be a killing blow right at the moment you need to accelerate. So these decisions early on are really, really important. And I think that's why developers gravitate towards solutions that are mature, things that they know that will scale, even things that are open source. You're going to pick something like PlanetScale for your database provider, not because it's the cheapest or because it's the most fun to use, but because you know it's going to be a durable provider that you can scale on for years.

1:23:33And WorkOS is like that for auth. At the earliest, earliest days, if you look across all these different services, they kind of look very similar. But at day 1 ,000 or day 2 ,000 or day 10 ,000, you're going to want to have made sure that you picked a platform that could scale with you. And today WorkOS is powering auth and identity and security and permissions for all of these AI companies, literally the fastest growing companies of the world, like OpenAI and Anthropic and Cursor and Perplexity, WorkOS is under the hood there. So I think when people pick WorkOS early on, really what they're doing is trying to pick the defaults to allow them to grow and rapidly scale.

1:24:07And there's no platform other than us that's done that at that same level.

1:24:11Jerod Santo:Well, friends, the next step is to go to workos.com, sign up today, check it out, free for a million active users. Try it today. There's no excuse not to. It is your default. You should choose it. So do so. WorkOS.com. Once again, WorkOS.com.

1:24:37Jerod Santo:Let's go back to, if you don't mind, why Python? I am not against Python by any means. That's an interesting question. But why Python? And let me frame it from this perspective. Let Let me give you my framing and then you can answer however your heart's desire is because I like Python, but I also like, you know, maybe on a network device, maybe Rust might be more pertinent. Like in today's age, Rust is all the rage these days. Yep. Or just mainly something that compiles to a single binary. Yeah. And it's easier to deploy and reason about rather than a complex Python code base and how you deploy it.

1:25:16Rust and Go will be the two things that kids will be calling for, right? And we've got skills in the company. Yeah. So we've got kids in the company who are hitting at us every day for being the old boomers using Python. And part of it genuinely does come from our background. And when we started, we were going for what can we use nicely? What can we reason about? What can we do safely? And we experiment a fair bit with new technology for new tokens. But really, what you go for is stability across our development base. If we were starting fresh, we could be convinced to say, hey, let's do this thing in Rust.

1:26:07In fact, we've got projects going right now that are distinct enough that are written in Rust. But when we started, so 10 years ago, it was the best alternative for us then. And it's not enough for us to move off it yet. Like we don't think there's significant reason to. In fact, we had to do a Python 2, Python 3 rewrite for some of our stuff. And I personally, like earlier you spoke about capitalism, you start to get to the stage where you realize you're old. And like for the Python 2, Python 3 upgrade, I personally hated it because we take a bunch of engineers who have to now work for months.

1:27:03We've got to make sure all our tests are correct. We've got to make sure all our edge cases are correct. And for that months of work, customers get zero benefit. Like they're getting the same thing they used to, except now it's in a different language. And so if the thought comes up of rewriting in Rust or rewriting in something trendy and hip, it would take a lot to convince me, mainly because you end up spending a lot of time on something that doesn't directly benefit the end user. So, yeah, mainly that's where we are. And for sure, like when you're hiring younger, hipper developers right now, like you'll attract younger developers coming in going, hey, we're Rust shop or we're Go shop.

1:28:01Python still doesn't have heavy negative connotations. Like it's not Java yet. But yeah, that would be the answer. It was good for us. it allows rapid iteration and at this point doesn't add a significant drag. But, but certainly, yeah, things like Rust and Go are pushing in, like, like they're not just trends, they're here to stay and they're going to bring goodness. But, but that's the reason for the most part.

1:28:40Jerod Santo:Yeah. Well, I just think about, I mean, sure, Docker solves some of the problem, right a dev container solves some of the some of the problem i think about a developer and maybe these are things you care about in terms of ergonomics uh because you're not a you're not you know the tried and true typical capitalist like you're actually trying to make a dent in the world and making a dent in the world isn't just you know uh fat in your pockets with cash it's like well maybe i can actually make this project a bit more approachable maybe potentially easier to develop against. And if you have the desire to want canaries everywhere and the largest beacon to your funnel is canary tokens, well, heck, why not, right?

1:29:23Jerod Santo:Just make it even more... Not that Python is not. So I'm not trying to hate on Python by any means, but it's... When you can consider deploying Go to a system, so much easier. A single binary. Systemd on any given Linux. A binary is pretty easy to reason about in a lot of cases. You know, you can, it could be an API, it could be a CLI, it could be an MCP server all in one single binary. And heck, if you want to be nefarious or anti-nefarious, you can wrap a little canary and token in that binary as you talked about already, you know? Yeah, no, it totally makes sense. And it's stuff we've considered.

1:30:02At this point, the honest answer is like 10 years in, like a legit software company, as we become adults, you end up with history and legacy that you carry with you. Debt, but not tech debt. Yeah. It's just debt. Yeah. And you have the debt of choices, but one of the things we keep trying to do is trying to make sure that we're still making good choices. And so it's stuff that will come up. And yeah, I'm pretty sure we'll see a token released soon that's actually Go on the back end. And so it's a Go token now in the same framework because again, yeah.

1:30:47Jerod Santo:That's interesting. And actually an actual Go binary as a token itself to like instantiate it like a Go D kind of thing or some sort of like demon that runs in the back. So that's what will be running in the background. What is this? Let me hit it. Let me, whatever the binary's name is, dash dash help. You know, let me sniff it a little bit. It should be exactly that. I'll give you a heads up when it's live so you can go. That makes a ton of sense. Yeah. Wow. Yes. There's so much you could do with this room. No wonder why you are so excited because I mean, your world is endless. Your world really is endless, man.

1:31:25Jerod Santo:Yeah, it is pretty good. And you're making enough money, which is phenomenal. And honestly, you know, my only the reason why I ask you this is I'm just such a Go fan. I'm just such a Go fan. I'm not like I'll write Python all day long in the place where it makes the most sense. And it's usually in the age of data or analytics or agents or it's TypeScript because I'm interfacing with an SDK or something like that. And I'm harnessing, you know, an API call to an LLM, you know, so I kind of hop into TypeScript and Python. Usually in those times, I'm happily becoming a BUN developer. And, you know, my choice of stack when it comes to web these days is BUN plus Felkit.

1:32:08Jerod Santo:like you put those two things together you can deliver a binary you can SvelteKit your way to a binary it's so amazing I haven't hit the edges yet where I'm getting bit by that choice yet but thus far as paying great fruits otherwise it's Go it's Go Rust when it makes sense when it's like network and I don't want any latency I want total performance it's got the garbage collector so you have those issues it wouldn't make sense to build a DNS resolver in Go You could do it, but you would pay the penalty of latency on network calls and that hot path of a DNS lookup. You don't want to slow that down with Go.

1:32:46Jerod Santo:You want to do that in Rust. So it's super interesting. So we've got like our network switchboard, which is our DNS server, runs in Python. And like I say, we push binary updates through that with lots of the heavy lifting done by Twisted. Thankfully. But and and we've just I think we've just got a Rust based DNS cache proxy written fairly recently for for some internal work. But no, it's goodness. And and we're not super religious about our tools like we'll use the best thing for the job. So yeah, for the most part, watch the space. Is your desire for Canary Tokens to be self-hosted by a lot of people?

1:33:44Jerod Santo:Like if you had to zoom out and say my Wave of Magic Wand wish list, not that I need it, but like if I could just have Canary Tokens go this direction, which direction would it be? No, if I'm super honest, I like people using the hosted server. And mainly there's a form of service level that we can give with that server. So we've been running that server for 10 years and it's been down only a handful of times. And if someone self hosts you at the, like, you don't know if they run it right or put it on hardware that gets filtered or something like that. So generally, I like running it on the server and doing it nicely.

1:34:35Our honest hope for it is that we can provide stuff that's so useful that people use it easily. Even with Canary, a counterintuitive thing that actually was a problem for us early on was people who overthought Canary. So it's like everyone thought this has to be complex and people would buy it and they're like, hold on, we just want to strategize about where to deploy it. And like eventually, initially we didn't have the confidence to say it, but we saw it happening enough times to say, listen, don't think about it. Or like, while you're thinking about it, just go plug these in. And inevitably, it would catch stuff while they were still deciding where to deploy it.

1:35:28And so - Like mysterious activity? Yeah. We've caught so many attackers during our POCs. So customers arrange a POC. We send them a canary. We set up a console for them. And during the POC, they catch attackers. Like we've got tons and tons of cases where this has happened. Because most customers just don't know how much bad stuff is bumping around their networks because they don't. Like the state of network defense is often surprisingly bad. And so one of our big things became saying to customers, listen, don't think about this too hard. Like, just deploy them. Like, the only way to do it wrong is for you to not deploy it.

1:36:17So you might come up with the perfect deployment strategy after you can, but for now, just plug them in. And again, I mentioned early about how Canary, like one of the things we got lucky with. I would have mentioned this on our first podcast, but for new listeners, when you think of like a honeypot or deception, your first thought is how you've got to make it really high fidelity copy of your network to fit in. and you start to think how you've got to craft this thing to hide in your network, right? And you'll end up with a really strange realization that we got really lucky with. So for example, take that AWS key canary token I spoke about.

1:37:09Like you'll get a customer who says, hmm, we're not an AWS shop. We're an OCI shop. Like, can you give us an OCI token? and we can totally do something for them or if they're an Azure shop, we can give them an Azure certificate. But here's the thing, an attacker who breaks in and is orienting themselves, who finds an AWS key on your network, doesn't have the luxury of saying, is this an AWS shop? I'm not going to use this AWS API key on this network because I don't think this is an AWS shop. They see that key and they're going to use it. So if you take our canaries, right, they can act like if you've got an OTA network, like they can act like a Simmons microcontroller.

1:38:06They can act and they can talk good Modbus. But in honesty, if you've got an OTA network and you just placed a canary that looked like a Windows file server on that network. Do you think an attacker who lands there and sees a Windows box with an open file share, they still have to hit it? Yeah, it's too tantalizing. They're like, what did I find? What that means is that you can't deploy a canary wrong because your initial thought is, I've got to make it look like the other PLCs on this OTA network. But the honest answer is, you could have made it a Windows box and it would still work. And so there's almost a thing where because security has had all this complexity for so long, you think it has to be complex to work.

1:39:02and actually just conceptually these things work while they're simple which we just got lucky with like we didn't know when we started but yeah that's how it is and it works

1:39:19Jerod Santo:so your desire for Canary Tokens is to be not generally self-hosted although you can there's no limitations but you just prefer it that way For lots of people to use it. And in honesty, I think most people, like when we make all our stuff internally, we fight very hard for, is this too much work for the customer? And self-hosting, there's a portion of the population who can and should, so we enable it for them. But most people, like, they want to go about their day and they want to get done with it. And so our thing is to make it easy enough and simple enough for all of those people to get the benefit anyway.

1:40:10And so you'll see almost everything we end up doing, we end up saying, it's so easy. Visit us, click this, wait till it alerts you. because we think there's a sweet spot there of people who could do a lot of things, but never get around to it. And instead we make that happen reliably for them. So if I'm honest, those are the people we most want to help because the people who want to self-host, they'll be able to do it. Like we make it easy enough so that they can. But I think the other people are underserved and we're trying to make that happen for those people.

1:41:01Jerod Santo:What about products that could bake? What, I guess you could bake a token in, But what if you wanted to productize inside of your own product the usage of Canary tokens? I'm thinking like auth providers or like some particular applications out there would be really interesting just to bake in. I can't recall GPL's license preventions. Do you recall what limitations are there? So GPL normally just says if they're going to take it and make changes, they must push those changes up. To open source. Yeah. So it follows our hippie roots. But guys who are self-hosting do their own thing. And people who want to extend tokens, because we publicly write about how they built, like you can build clean room implementations of it just based on like with many of the tokens, there's the initial, I say clever because I'm talking about our own stuff and everyone thinks their stuff is clever.

1:42:14Like there's the initial trick that says, here's how you can get a signal out of this thing that something's happening. And once you know that, you can build it a bunch of different ways. And so when we write about it, we make that possible for people. But yeah, we'd also be happy to play with people. In fact, last week, the author of Santa, which is application, open source application listing, app whitelisting and ignore listing, they pinged on Twitter to say, hey, the same thing that we did for the CloudStrike token. Can we do something with them so that their API keys are canary tokens also?

1:43:06and yeah, we'll be happy to play with them and do that sort of thing. So yeah, there's room for people to play and other vendors to play. We'll happily work with people to make that stuff happen.

1:43:21Jerod Santo:Yeah, because I mean, even on my, I mean, I'm just a little tinker here, but I am building some stuff and that stuff may get used by lots of people. It may never get used, who the heck knows? But I'm gonna build it like everybody else is and I like to know my limitations. and one, I'm going to play with Canary Tokens. I'm glad we had this conversation to remind me that this exists. I'm glad we went down the road of the fact that there's a code base behind this and it's open for me to tinker and play with because I'd like to just, just for the fun of it really, consider tokens that matter to me that are not tokens that matter to you in the Canary Tokens world so far.

1:43:56Jerod Santo:And I'm sure there's some. Yeah, it makes perfect sense. and look, if you're doing any of that stuff, drop us a mail, like the team. We like our stuff getting used and we like cool use of our stuff. So drop us a note and we'll be happy to hit you with thoughts and jump in. Yeah. If I sent a pull request that said, rewrote this and go, would you be upset about that? That's just a joke. Is this a joke? It's just a joke. It's just a joke. Like, hey, what is this PR? Like, everything is gone. Okay, gosh. Because, I mean, just because, just because. I'm not much of a Python guy myself, but it's okay.

1:44:41Jerod Santo:I will navigate your world just to play. Your agent will be. Don't worry. Yeah. Well, I just think even, like, where's the playbook for what is a canary token? like is there a specification for a canary token that i can easily read about in the docs i imagine it probably is yes yes this is how my brain thinks now you know it's like yeah so so if you hit docs dot canary tokens dot org um you'll get a whole bunch of documentation on how to use them what they're doing. So docs.canarytokens.org would be a starting point. And there'll be a whole bunch of stuff for you there. Do you think that this is the biggest beacon to say things exist, canaries exist?

1:45:36Jerod Santo:That's a good question. And like, since you don't have this desire to market, like what if you marketed through just the prowess of your awesome software in a way like if you made just by sheer will of force being that cool canary tokens not that it's not cool but cooler because everything can entropy right everything can always be cooler and better and faster and stronger right so like if you put more weight behind the the the awareness of yeah and not even because you're marketing it just because it's just got so much gravity coming to it that naturally your kind of core business gets exposed and more people learn about and use canaries i think it's certainly possible like i have a uh and and again probably not a great answer there mainly my my answer there is that so far, like, yeah, I don't know.

1:46:35When I say it, it sounds terrible. My intuitive answer would have been that we're pretty happy with our growth. Like, there's a line to be walked between like customers who pay for the stuff, canary tokens that we build. And right now, we're pretty comfortable with it. Like our client base keeps building, Canadian tokens keep building. I'm trying to think if we had a massive number of people building for it, whether that would make a difference. Yeah, I guess it wouldn't hurt. Mainly, I think there's an interesting thing where, like when we started, the thing on our mind was like, can we build a thing that's useful?

1:47:31Like, can this concept work? And this was Canary. And then like, we had some people buy us and then we said, okay, like, can it work well enough that they'd renew? And then our thing became, okay, but can like customer X like this? And like at this point, like genuinely, our customer list is crazy, right? Some of the best security teams in the world have blogged about how Canary saved their networks or what they do with Canary. And so for the most part, what we want to do is keep that going. And we've got a pretty firm belief that if we keep doing that, that stuff keeps radiating outwards. And I'm not particularly worried about the VCs talk about like, throwing jet fuel on the fire.

1:48:34And like, I don't think we have to throw jet fuel on the fire. Like, like, I think, that's not my suggestion either.

1:48:41Jerod Santo:Even let me help you understand the lens of my question. is less on the suggestion that, okay, because canary tokens are cool and it creates this natural visibility, it's not the lens of growth. It's the lens of more people using this tech because it's so useful in what it does. The breadcrumbs idea you got is just fantastic. Obviously, the idea of canaries are fantastic, as you mentioned how you've caught some in POCs. it's more like the desire as a maker and a builder to see a useful thing be used be used that's where my lens is coming from yeah i i understand and no like like for that getting it used by more people is a win like like it warms our heart like it's why we're doing it so so more people using it uh is a win and yeah we could like i guess there's there's more to be done with uh marketing it like uh and and getting it out there um yeah it's it's worth well sales and marketing are two different things and that's why you always have the phrase sales and marketing sure uh marketing largely is story yeah and usually people leverage that story for sales sales right but you can leverage that story for just the fact that you're cool you know yeah just because we're cool we can just the same reason why you put effort into the hoodies that you do that you use quality hoodies versus something that's not high quality sure you put care and touch into everything that's kind of where i'm coming from yeah because it makes it makes perfect sense i think there we probably uh stay too close to our early uh We'll build cool stuff and it will get out there.

1:50:37And there's more work we could do to share that love. And it would mean more people using it and getting saved. So it's worth taking away. I'll speak to our CEO. Okay. Good one.

1:50:56Jerod Santo:Before we go, can you give me the juiciest canary story you got? What is the juiciest story? Like if we have CISOs listening to this or somebody who's adjacent to their CISO and they're like, my gosh, I listened to the coolest developer podcast. I heard from this tech I'd never heard of. They had the coolest views on their tech, the coolest views on revenue, how they run and shape their company, et cetera, et cetera. Oh, my gosh. They proved a concept inside a company and it saved them from X. Like just give me the juiciest story you got. The juiciest. So the first thing I'll say is people should check out a site called canary.love.

1:51:39So canary.love is just a collection of people tweeting nice things about Canary. And mainly it's from defenders when Canary saves them. But some of it is from pen testers and attackers saying, Canary caught me. It's the only thing that caught me. Here's why you should use it. Um, the best canary. So, so first I'll say, we seldom have a week go by without a post on our internal slack from someone who caught something. So some customer mails in to say, you caught our pen testers, or you caught our attackers, you caught this. So, so like empirically, they just work. But some of the best stories we hear about Canary, we only hear like when we're at a conference or at a trade show and someone comes up to the booth and whispers to us because then it's like, hey, this happened and police were involved and we kicked down the door and the stuff went down.

1:52:45But one of the things we didn't expect was how heavily canary tokens would end up being used by law enforcement. And so law enforcement all around the world end up using it to track child sexual abuse cases. And so we end up with a whole bunch of love From police departments saying This is what the stuff is used for Here's how we catch people And for us it's been Like genuinely there's making an impact And there's making an impact And like we've gotten mails from police departments saying We've been after this dude for so many years And this is how tokens helped us track them down. And thanks.

1:53:42Yeah.

1:53:43Jerod Santo:These breadcrumbs would be cool too. Yeah. So that stuff has been amazing. And look, like I said, I don't know if we ended up covering it, but like really big customers. For us, there's still an element of it where you're still a kid. And like if NVIDIA is the hottest company right now and they blog about embedding DNS tokens inside of models to detect when they stolen or Grafana, who we use internally for some dashboards, put a blog post saying, we told you we caught attackers, but how we caught them was canary tokens. Like that's the story I want to hear. Give me those stories. What more can you share about NVIDIA and their DNS token and models getting sold into – Grafana's a friend of ours too.

1:54:33Both of them are just public blogs. Like NVIDIA let us know. So I'll send you the links. But NVIDIA basically told us, hey, we put this blog post out and we talk about using canary tokens to protect models. And Grafana had a super cool blog post because they basically wrote a blog post. I think they called us their unsung heroes. or something like that, which was super cool. And they basically gave a how-to on Canary tokens because they ended up catching attackers on their networks through Canary tokens. And so that sort of stuff, yeah, it absolutely makes it all worthwhile, right? Like it's happy customers saying stuff we didn't even know they were going to say and giving real-world stories about how they caught attackers.

1:55:26So, so for us, it's perfect.

1:55:30Jerod Santo:I guess last question for you, I think, is are there any canary tokens in the canary token database? Or sorry, are there any canary tokens in the canary tokens code base? They are not. We have them all throughout our infrastructure, but not in the canary tokens code base. So even canaries are canaried. And so if someone were to hack into the devices or if people were to hack into our infrastructure, like we eat our own dog food. But no, canary tokens is what you see is what you get. And it's pretty straightforward. That's a really interesting idea to figure out how to sprinkle. And I guess you'd have to be, there's certain cases, maybe like a Palantir, you know, not an Adam Tinkerer code base where it matters, but like a Palantir kind of code base or even Claude with, you know, the bombings in Iran around AWS data centers.

1:56:36Jerod Santo:Like they're going after, like you can't write this headline, bro. The data center is getting, I mean, the saddest stuff in the world, obviously, but data center is being bombed, not just because they're data centers, but because Claude exists in there. It's crazy. Like seriously, like did Predator not – or not Predator, Terminator not predict some version of this? Like you can't write that headline where you – a data center is not down because of a key role accidental. It's down because of a bombing because Claude is hosted there. It's insane. You know what I'm trying to say? Yeah. Gosh. No, it's, but, but look, literally like tokens and, and canaries, like we see deployments like that.

1:57:26And, and we've, so the cool thing about it for us as a technology that we've built, like I say, it's, if you're a two man law firm, like you buy it and plug it in and don't do anything. But if you NVIDIA on your technical blog, you say how you're embedding them in a model. And if you Grafana, who are super technical, and you're saying, this is how we're deploying these things and how it catches attackers for us. So it allows people to just use them and it works. And it allows people with super smart security teams to extend them and get value. and mainly the thing is you start and then you figure out how you can do more with it.

1:58:18Yeah, I like this.

1:58:20Jerod Santo:I mean, every time, I mean, every time we've talked, this is twice now, but I think about our conversation a lot from a couple of years back and I've been a fan, I'm smarter now, and I can see how I can actually deploy these things realistically. Like I've become a better networker, not like physically, like as a human being, but like, you know, software networks. Yeah, I've learned a lot. I've steeped, you know, my fascination is in the home lab. I learn a lot. Like the coolest thing about the world we live in is I'm not one of those home labers where I'm like, I got to self-host all the things just because like, like really my home lab is where I experiment.

1:58:59Jerod Santo:It is certainly a laboratory for me. It's an exploration. It's a map of where I could go, where I want to go. And that's what I love most about home labbing. I learn about networking, learn about DNS, learn about storing and hosting data, learn about propping up VMs with Proxmox or I don't know if you've touched this yet, but Incas. Have you played with Incas by any chance? I have not. Are you a fan of Linux containers? Yeah, sure. Containers, obviously, right? Docker containers? Well, I keep mentioning this because I'm going to have the person on the show. At some point, Stefan, he runs this project.

1:59:37Jerod Santo:It's a fork from Canonical. You know Canonical? Ubuntu? Canonical had a project called LXD. It had some licensing changes. Some of the original creators and maintainers of it weren't – I don't know the exact story, if it was bad or good. But just for whatever reason, it's a fork. Right. And it's not – it's a derivative fork. So I think LXD went – I think it was something around the – when you commit code to it, having to sign an agreement and not truly open source contributions anymore. I think that might have been some of the details behind it. We'll clear this up at some point in the near future, and I'll stop paraphrasing this jacked up story I have, at least my version of it.

2:00:19Jerod Santo:But Incas is a fork when it was open source of Canonicals LXD, which is Linux containers, it's system containers and system VMs. So kind of like one layer below what Docker is. It's not quite. It's similar. You can run a VM like that. Like a VM is like a Docker container in a way. Yep. If I understand correctly. Maybe I could be wrong. But Linuxcontainers.org, Incas, it's the newest hotness out there. It's so cool. And I think in your world of what you do, it could certainly be part of your stack and what you are building over there. Canonicals, another South African company, kind of. Kind of.

2:01:03Kind of. Well, yeah. So Shuttleworth was a South African technology guy. And then he went and took, made Ubuntu happen. But yes, we just claim him for South African heritage.

2:01:18Jerod Santo:I mean, my distro choice for any given Linux server, hands down, is Ubuntu. I mean, love the tech. That's what he did, right? Like he made a Basquillion selling his tech company in the early 2000s. Yeah, like it was the crazy early days. I love that term. He used it to go to the space station on the one hand. And the other big thing that he did was he, I think he forked Debian and put money behind it to make Ubuntu. And yeah, it's been great. Like I think it's been a genuine gift to the world. Yeah. Well, since you mentioned his name, Mark Shuttleworth, that's a fellow I want to talk to. His first name is Mark, right?

2:02:06Jerod Santo:Yeah. Is my memory correct? I thought so. A lot of names in this tech world. But that is, if you're listening to this, Mark, much love, man. Come on the show. Or if you know somebody who knows somebody and you got away, I'd love to get him on the pod at some point. I'll fly to you. You can fly to me. We'll figure it out. I live in Austin, by the way. It's a lot of good barbecue. I'll attract you with barbecue and sit down with you on a good podcast. But big fan of Canonical. I mean, in every capitalistic nature, there's a moat to protect. And I get that, right? And there's business to be done.

2:02:37Jerod Santo:And I get that, too. Sometimes it feels like a rug pull in the open source world, and sometimes it's just business. And that's okay, too. That's why we have licenses. That's what defines the line but doesn't define what it is. It defines the line of open source. There are particular licenses that define the line of open source, and there's some lines that get crossed, and it's now that source available, and there's some lines that don't get crossed, and hey, that's truly what we call open source as we know it, which is fine. but big fan I just mentioned Incas because it's cool I don't even know why I brought it up in the first place totally cool, it got us to the shuttle what plug, which is worth it yeah, for sure for sure, for sure well, in closing here, Harun what's left unsaid, what did I not ask you that you're like gosh, you know what Adam, how do we not name drop this or mention this, we've got all the URLs, we'll scoop them up and put them in the show notes of course, but anything here in closing that we can say and make sure we get covered No, I think that's it.

2:03:37Have people check out canary.tools, canary.love, canarytokens.org. And yeah, if they have questions, hit us up on Twitter X or on Mastodon and we'll happily hit them back. But yeah, thanks for having me on again. It's always cool.

2:03:58Jerod Santo:What is your Twitter handle? Oh, here he is. ThinkstCanary. yeah so at things t-h-i-n-k-s-t canary that's the one i'll put that in the show notes y 'all don't you worry about that thinkst canary on x twix call it what you want and then harun mir his full name put that in the show notes as well follow harun this is cool stuff harun thank you so much for coming back on thank you for uh just sharing the sharing the depths you know being so focus on creating cool tooling and not just a fat pocket of cash, but, you know, employing cool people, doing cool stuff, putting it out there and being so willing to share.

2:04:40Jerod Santo:Really appreciate that. Thank you. Thanks, man.

2:04:46Jerod Santo:Well, friends, thank you for tuning into the pod. It was awesome having Haru Amir back. One of our favorite people around here at Changelog. Fingst, a very cool company, 50 people, $22.5 million in any recurring revenue, no price increases, all inbound sales, and they're doing cool stuff. That's gotta be the coolest story ever, honestly. And seriously, I would love to just do nothing but play with their tech. The breadcrumbs, the canary tokens, and it's open source. All right, big thank you to our friends over at Buildkite. A big thank you to our friends over at WorkOS. And of course, a big, big thank you to our friends and our partners at fly.io.

2:05:25Jerod Santo:And to the Beat Freak in residence, Break Master Cylinder, bring in those beats that we love so much. Thanks again for tuning in. This show's done. See you next time.

2:05:50Jerod Santo:Win it. It really whips the. All right. That is the show. I run. Thank you so much, man. Was super elite. Thank you, man. I have an idea. Actually. Yes. Do you have, could you spare two or three minutes more? Yeah. Yeah. Okay. Yeah. I'm going to, this is for our plus plus audience. I almost forgotten about asking you this in the main show. And maybe this won't go down well, but maybe it will. How would you feel a, about a more capitalistic, let's just call them a reseller. Yeah. Distributor of things canaries. So maybe you're not desiring to grow more, but what if I knew the ability and I built my own company that only sold, deployed, serviced, loved your canaries?

2:06:44Jerod Santo:And.