#712: The Frugal Architect w/Werner Vogels: Watch Duty Keeps it Simple to Save Lives

0:00This is episode 712 of the AWS podcast, released on March 17th, 2025.

0:09Hello, everyone, and welcome back to the AWS Podcast. I'm Les here with you with a very special episode. This is another of our really interesting series called The Frugal Architect, and there is no frugal architecture. Without that, man, you know well, Dr. Werner Vogel, CTO of Amazon.com. Welcome back, Werner. Thanks, Simon. Thanks for having me. Always a pleasure, and we have some special guests here today for what promises to be a pretty engaging conversation. Firstly, the CEO of Watch Duty, John Mills. G'day, John. Thanks for having me, Simon. Excited to be here. Good to have you here. And of course, wouldn't be a frugal architecture discussion with the CTO, David Merrick.

0:49G'day, David. How are you going? Hi. Nice to meet you both, and thanks for having me. Awesome to have you guys here. Now, we are talking about a topic that is interesting to people around the world because it's about scale. It's about safety. It's about doing things on the smell of an oily rag, as we would say, doing things frugally and simply, but also things that are critical. And often people conflate criticality with expense, but often criticality breeds innovation. And so, John, maybe let's start at the start, if we can, of what is Watch Duty and how did it emerge? Yeah, I mean, simply put, Watch Duty is an emergency alerting platform.

1:30similar to what the government rings your phone at, that very loud noise at random hours of the night. We're an organization that is providing emergency alerts regarding wildfire. And it was born, unfortunately, out of fire. I live off the grid in Sonoma County, where we experienced this at a regular cadence. And after a couple of times, you start to realize, like, who's going to tell me when danger is nearby and at what interval level. And after a while of living through it, it just decided to do something about it. So how are your customers, John? Well, our customers really are everybody. It started out as a application for citizens because we assumed that the government themselves knew where everything was at all times.

2:20And as we've scaled, we found out that tanker pilots, dozer operators, emergency managers, firefighters, and everybody used it. So it was that classic case of you built something thinking it was going to be used one way, but customers get a choice, don't they? They choose what they like and what they want to do with things. And this went well beyond what you planned. Yeah. I mean, it definitely is used for the same use case, just a different user base than we imagined. We didn't realize it was literally everybody so i understand it started off as a volunteer effort because it's also a bit about it yeah you know dave was a volunteer uh here and so was i right but uh dave and i were working on it for almost what 18 months dave i think and then you then you came on board after that um but it's still a lot a lot of volunteers yeah i was gonna say it was two two full fire seasons um i think of them in fire season years really i guess it was less than 24 months um i mean it was such an interesting start because i've never you know i've done you know hackathons and sort of four good projects before but this one so quickly caught momentum and so quickly had a fit in the market you know you talk about customers and competition the weirdest thing here is that it filled such a gap that there was no competition.

3:43You know, like no one else is doing this and no one else, it's not like we were an incremental improvement upon something that existed. It was, okay, you can go on Twitter and Facebook and have 12 tabs open for your life and safety, or you can use a simple app where, you know, experts are synthesizing this for you. It's like, it's such a, it was such a gap in the market that there really isn't competition in the classic sense. Well, I think it's interesting because one of those use cases, because we, of course, we don't call them wildfires here. We call them bushfires in Australia, and we're very familiar with those.

4:19But it's a classic seasonal thing that happens. And so for most applications, seasonality is not a good business model because particularly when it's seasonality of not, you know, oh, it's really busy because it's, you know, gift-buying season, now it's just normal buy. it's fire, no fire, fire, no fire. And the no fire time is where people sort of lose attention. And like you say, when I look at your app and the capability of it, it's like, wow, how didn't someone already have this? And it sounds like you were in that situation where you sat there, but you took that next step of action of saying, well, how does this not exist and how do we make it happen?

4:56What helped you make that jump? Like, what was it that, you know, We can all see something, oh, well, someone should build that. But you built it. Yeah, I mean, again, after experiencing a bunch of fires, two of them being a quarter mile from the edge of my property, you start to realize that someone has to do something about it. I live off the grid, so my nearest neighbor is about a mile away from me. The nearest fire department is six miles, I guess. And so you start to realize when you make your own power, you make your own water, you live on your own infrastructure, like you're on your own.

5:31And so I could sit there and say, well, who's going to do this for me? Or I could start doing something about it. And luckily, Dave was crazy enough to join me early and help me architect this thing. So tell me a bit about sort of what, how does, I mean, in Australia, Simon, tell me if I'm wrong, There is a user app where individuals can actually signal, oh, there's a fire here, there's a bushfire here, or it's starting here. But that's not what you guys are doing, huh, John? And Dave, this is much more where you take professional information coming into the app. Yeah, it's really more expert-sourced than crowd-sourced, right?

6:15So like you can go on Citizen Next Door Facebook and start screaming in the ether and tell someone that you see a fire, but is anyone going to hear it? You know what I mean? For us, it really starts with someone calling 911. And then we hear those dispatch signals. Our volunteers and some paid staff turn the radios on and start listening. So they know what to listen for. They know what these words and terms mean. and they're really synthesizing this information on the fly to make sure it's not literally the boy who cried wolf. So they are professionals or former professionals. They know how to interpret the signals that they're hearing on the radios.

6:56Yeah, it's kind of a mixed bag, right? Some of them are kids who grew up with a scanner in their house because their dad was a firefighter and they're literally worried, is my dad going to come home or not? And then some of them are the dads who've been fighting fire for 45 years and now we're tired and they can't turn the radio off. And so like once fire is in your blood, these people can't stop. It's a life of service. So let's talk briefly about, I guess, what you reached for when you went to build this and how big is the team building this piece of software? I reached for the pieces that I knew well.

7:29I went through the classic, all right, well, let me draw up a diagram. Let me write some things down. I wrote all these options for where are we going to host? What kind of database are we going to use? And my mind drifted like any shiny object engineer to like, well, it'd be nice to learn a new thing. And then I thought about it a little more and I realized that was a terrible idea. You know, maybe, maybe if it was my full-time job, I would have had the time to invest in, you know, learning a new language or architecture or database or really anything new. So every decision ended up coming back down to things that I was really, really familiar with and that generally were very, very simple, boring choices.

8:13You know, I think a lot of people agree now that like you only get to spend so many innovation tokens, whatever you want to call it. You know, you get to pick a couple things that are wild or innovative or novel. But in this case, it was like, well, everyone has a job. Everyone's doing something else. We need to pick the absolute simplest things. So we're going to pick, you know, the technology stack I was using at my current job and, you know, all the things that were already just the easiest panel. At the beginning of it, you must have been thinking about, this is a volunteer effort, but how much is this going to cost me out of pocket myself?

8:52Did that drive any of your early decisions as well? I was the only one who didn't have a job at the time. So I was, I was writing, this was my job. I was writing a lot of code. And so, you know, we picked Django, we picked Python, we picked React and things that we knew we could pick up off the street to find volunteers who could do this rather than, to Dave's point, the newest shiny whatever would be super cool. But like, guess what? It's me in the woods by myself coding while Dave and others are volunteering at the time. And so it wasn't going to work very well if we didn't pick standard, you know, standard practice types of things, including architecture like AWS and others.

9:32and and then just to go to the cost you know it's easy to look back now and say wow nine million users in in the first two weeks of january good thing we picked cost-effective choices you know we didn't know at that time what it would look like you know it was amazing how much pickup it had that first year but we ended that first year with just shy of a hundred thousand downloads i think um and that seemed crazy you know still not that expensive to serve and in what we're doing, mostly read heavy, you know, API traffic. But, you know, we didn't even quite realize at the beginning what the scale of that cost might actually look like down the road.

10:13It was more like if we actually get there, we can fix it then. And this is often a classic case of, you know, I'm building something to prove it works and then I'm wildly successful. But often the timing of the wildly successful and the building it and then having the time to make it better doesn't line up. And particularly in your case where this thing went ballistic in terms of adoption, I think it was one of the top, if not the top downloaded app for a number of months. Again, some of the names you'd think would be the most typical ones that are always downloaded. So you're sitting in a situation where it's a small team and we need to get into the composition of your team because I think that's fascinating as well.

10:51But you've got this small team that suddenly doesn't want to have to be in complete, dare I say it, firefighting mode on the app. You just need the app to run. So having the elasticity is, I mean, that's when it makes all the difference, doesn't it? Yeah. And I think, you know, we did understand that we needed to be able to pull the slider to the right if we needed. And, you know, if we really needed to spend money, we could all just find, you know, the cost to do so. but again I think it we also knew that there would be time so you can go back to the seasonality normally a terrible thing in software businesses but it was actually especially in the first few years it was a it was kind of like the dream on the engineering side because you actually got periods of time where well we had no product organization we had no sales no one was breathing down our neck to do anything besides to improve the application so you actually had these slow periods where you got to fix things, where you got to focus on performance, where you got to do that work that engineers fight so hard to like slot into the roadmap.

12:01And it was like, well, here's a couple months where at the end of these months, all of our libraries should be upgraded, all of our performance hotspots should be dealt with, all of that technical debt that you naturally accrue with a small team, with quick decisions. We actually had those periods of time to really focus on it. That's slowly going away, but I still think we're trying to keep it in our approach and ethos because it really does help with long-term stability. Now, in that sense, I'm not saying it's as explosive as yours is, but retail in some sense, If you think about Black Friday and the month before Christmas, January, February, March becomes a time to fix things.

12:49But unfortunately, in retail, things never go back to where they were before. But you need to automate most of that. You've got a significant problem. Absolutely. Yeah, it's not a good thing. Tell us a little bit about the team. How big is the team? What are some of the talents on the team that make this possible? Yeah. I mean, you know, I don't actually remember how many people helped volunteer. I would say in the order of 15 that first year on the engineering side, 15 to 20 volunteers. That's it last. Varying levels of effort, maybe like five to 10, like really core contributors that first year on the volunteer side.

13:30And that second year, probably the same group. um and then you know it's it's a very unique composition because we started out fully volunteer for the first let's say two years and then we switched to a model where well i joined full-time um you know cto of one you know quite the title and then hired a contractor as i realized i needed a little extra help and really that whole first year was just um just myself until we started hiring a little bit more that second year and now we're at um seven seven engineers so still the one the two piece of team um the nice thing for us right now is because we had such a strong group of contributors we're able to recruit some really really talented engineers from from that group which is quite the quite the interview process you know if you've worked with someone.

14:26And, you know, in terms of how we're composing our team right now, you know, we still don't really know all the things we want to do. So we've been just trying to hire really, you know, that's an overused word, but full stack engineers that can are really focused on problem solving. You know, I don't think we don't have a lot of novel technology that we're building, but we are solving very, very concrete and real problems. So we're trying to hire and get engineers that love to own a problem from the beginning to the end. You know, there's not going to be someone in product building out all these requirements and then a deep technical dive.

15:07It's like you need to be the person that loves seeing an issue and then seeing it all the way through. And the nice thing is right now we have a really, really experienced team, a lot of talent and density. So we're able to get a lot done with a really small team. How did originally your volunteers show up? Were they people you knew? Were they early users of an app? Were they, I mean, how did you manage to get five people on board to donate their time? It's a problem that most nonprofits have where they only have two people that are maybe paid, but rely on a lot of volunteers. How do you find the people that are willing to donate their time to you?

15:50Yeah, I mean, we found them pretty early on through, mostly through our networks, through mine and Dave's. I mean, a lot of the people that we brought on board were friends of ours. Dave and I worked together at multiple companies. I have a couple other friends who I've worked with or known for a long time who cared about this problem. We told a good story. We had a good product. I mean, our first rev took, I worked day and night for almost 80 days. I built probably 80 % of it. And then Dave and others did the stuff either AI couldn't do, or I had to sleep at some point. And then after a while it picked up, people started to see it.

16:24They knew what it was. They were using it. And that makes it much easier to then recruit people because unlike a lot of nonprofits, they often build things for other people that they have never experienced those problems, which I think is really hard. I'm just going to add in that John is very good at convincing people to help him. So he's selling himself a little short there. Well, it's interesting too that you often call yourself sort of a tech-first non-profit, which is kind of a different profile than we're used to. Help us, I guess, nailing onto that and what it means to have, here we have a CTO and a prior CTO running this thing.

17:04How does it affect the culture of the organization? Well, it really comes down to product. So it has nothing to do with the tech stack, right? That's first and foremost. Like, who cares if it's peanut butter and jelly, duct tape and bailing wire? Now, can it take 100 ,000 requests a second like ours just did with a tiny team? Fortunately, it could, thanks to AWS and Heroku and other providers who allowed that to happen. But, you know, that said, I mean, this is a really organic product that is made to solve a problem. And so his first revision is essentially a simple news feed with a geospatial map, right?

17:40None of this was all that complicated. So if you kind of like strip away everything you don't need and just give it what it's supposed to be, like, it's a lot easier to get something off the ground, to think clearly and simply and realize we don't need complex engineering to solve these problems, right? We need a very elegant solution to a very annoying problem. And the answer was actually using human radio operators to solve it. The tech came second. And so there's a lot, quote unquote, better products than ours out there in terms of, I don't know how you want to measure things, better code or whatever it may be, but we have better outcomes, right?

18:18Outcomes are what matter. No one cares what it's written in. And so it really comes down to first principle product. I mean, I just want to add on, I think you're underselling our tech a little bit, John. Early tech. Early tech was pretty simple, man. You can see John's become a CEO. I think you look, everybody in LA just downloaded Watch Duty, and they got to see what Watch Duty looked like after four years of revisions. I think that's the big difference between a tech-first nonprofit and a nonprofit that wants to do something like this. because we set ourselves up to iterate very quickly to be able to take feedback from the SMEs, from our community, from ourselves, from everyone that has every single support ticket that's ever come in we've looked at and reviewed.

19:07You know, we've probably, I don't actually know the number on this, but we've probably released our mobile applications upwards of, you know, Upwards of 50 times a year, you know, that comes out to 200, 300 times. We've released, you know, we're up in the thousands of PRs and all of our back end and front end repos. You know, the ability to iterate quickly and have a, I think, a world class product on the tech side is that other difference. Because you can want an outcome and you can, and this is outside of the kind of scaling thing where 9 million people using your app doesn't tank it. But I think even on the product side, we've been really focused on iterating and bringing, you know, really, I think what a lot of people consider just normal product development in the startup and tech world to a user-facing problem.

20:02You know, a public-facing problem that a nonprofit is the best suited to solve and provide. You know, this wouldn't work if we're trying to charge people for this data. you know we we even if we didn't believe that everyone should have it i think we never would have gotten the traction um if we were trying to to profiteer off of people's most you know panicked moments maybe it would have gotten somewhere but um i think i think that's the big difference so can we um so we talked a little about customers with in essence to users but you know if you normally would build a product you would work from your customer backwards like They say you care about the outcomes.

20:46But there is a massive input stream here as well, because without the input of the volunteers that are actually putting things into your system, this wouldn't work. So do you consider them to be another set of customers? What do they need? What do we need to give them to be able to get this accurate stream into the application? So I think stepping back is, you know, it's when we found these radio operators, they were using Facebook and Twitter. Right. So they were using a very lackluster product to solve a very complex problem that was involving life and death. And so the bar was very low. Right.

21:31So obviously, four years later, we built all sorts of advanced tooling and civil noise processing, whether it's AI bots and whatever it was. but like we were immediately better than what happened before them. Right. And so we had leaps and bounds, better ideas about how they should be doing their job. Like, for example, they didn't even really communicate with each other. They kind of message each other in DM and Facebook or whatever it may be, but they kind of be running their own independent Facebook pages or Twitter pages. And they kind of know each other and like follow each other, but they never talked in real time in Slack.

22:07So imagine like four people covering one fire or like the Palisades fire, for example, which was like six to 10 people at a time. They're all collaborating in real time. So just the fact that we put them together in one place and gave them a community to talk and collaborate was a huge leg up, let alone all the systems that we built on top of it. And so again, boiling it down to like how things were versus how things are going to be, it was not as challenging as you may think. Now that said, as Dave's going to chime in, we've spent a lot of time building tools, right? But ultimately, it was so much better than it was that it became much easier to get them enticed in this and asking for features and ways for them to work better and faster.

22:52Yeah, I mean, it's a unique organization. You know, it is very much in one side a product-facing tech solution, but it would be nothing without that input stream you're talking about. And it's incredibly important. It's the secret sauce of what makes WatchDuty so compelling to everybody. And they are very much our customers in the other sense because, you know, there's only so many of them. It's hard to find these people. You know, it's a very specific domain expert that wants to spend their time and energy and really it's a passion for most of them. So we're trying to give them the absolute best tools that they can have.

23:34We had some volunteers that were also reporters on the engineering side. They were incredibly motivated to help fix these problems. We have advocates within their side that tell us, hey, this is busted. This we waste time with. Sometimes we just watch what the reporters do and say, you've been doing it that way for how many months? That's a lot of time wasted. You know, we had a, there's an anecdote last year where we found out that all the reporters had to refresh Facebook and Twitter pages all the time to find out how to report on more kind of like the end, the tail end of fires, you know, cause that's unfortunately where a lot of public information is posted right now.

24:16So they just had like a panel of 20, 30 tabs and they'd go every 10 minutes, they'd go refresh every single one. And it was like, what you're doing is super valuable but that is not valuable we can help solve that problem so give us a couple months next spring you won't have to deal with that anymore um because that is definitely the hardest part of our scaling is is the reporters and the contributors and that human element um which is i think the quality of data really relies on them much more than than anything else But by the way, can we do one little sidestep? Because I'm actually just curious in it.

24:57You mentioned Geo, John. What I find over time traveling the world and go to many different places is that commercial maps are not that terribly accurate when it comes outside of cities. And organizations like Humanitarian Open Street Map and the Open Street Map Group and things like that, what kind of mapping technology are you guys actually using because this is mostly off the grid as you mentioned john yeah it's a good question i mean we've god we've done so much experimentation with you know different tile sets from esri map box open street maps and honestly we still struggle with it you're you're hitting the nail on the head is like so out in the wild lands as we call them like there's oftentimes like satellite passes that are two years old and And you're like, that house burned down.

25:49That water tank's not there. Like, that's a dirt road that's now closed off. Someone bought that land and it's gated and they have no idea. And so we still struggle with that, honestly. And we're trying to find answers to those problems. And we're definitely not doing Google streetcars ourselves or, you know, helping open street maps, although we wish we could. We have no way of doing that. And so we're trying to think of other clever ways of doing that, whether it's buying satellite that imagery and partnering with those organizations. But there's a lot to be gleaned by the Earth observation community, which still hasn't seemed to solve this problem either.

26:28And I scratched my head living out here. I'm like, how did I just get here? And I'm seeing all these issues. And we've been able to open the can of worms and solve a bunch of them, but there's more underneath them. And this is a big one that we don't have a clear answer to yet. I mean, it's a big challenge, but I think, you know, part of the challenge and constraints that we have is we can't bite off all these problems. You know, we get a lot of user reports around, you know, this street is named wrong. This street isn't there. This satellite pass is old. And like, it takes a lot of discipline to be like, we're not going to build a middle layer where we start overriding OSM data.

27:09It's like, we have to be really focused on our core competency. and sometimes just be like, well, I wish that our product could be a little better on this aspect. I wish we could do it. But we're a team of one. We're a team of three. We're a team of five. It's enough. We could work all day and night and just distract ourselves from the actual mission. It's frustrating to have the street behind you mislabeled, but it's more frustrating to not have the product that we brought you. It's mostly because it is a massive problem. I mean, India is a really great example where, you know, many of the roads are not either not, you can't drive a big truck over it or a Google Street Map van or whatever.

27:52So there's this one app that actually is for Tuk Tuk drivers to connect you with a Tuk Tuk driver. And they in general go out into the woods. So this app now also actually tracks that data. Yeah, where did the Tuk Tuk go? So now we kind of know that there is a path to that. Which brings me back to actually another question for you guys. You collect a lot of data. You get a lot of input signals from all these professionals. Is there something that you do, let's say, with that data after the fire? Because this must be an extremely important data set. That is also for firefighters very important.