#730: The Frugal Architect w/ Werner Vogels: At Too Good To Go, Practical Engineering Keeps Food Out of the Bin

0:00This is episode 730 of the AWS podcast, released on July 21st, 2025.

0:09Hello, everyone. Welcome back to the AWS Podcast. I'm Les Heavitt. Great to have you back for another episode of our special series, The Frugal Architect. And you can't have that series without the gentleman himself, Werner Vogel, CTO of Amazon. G'day, Werner. Thank you, Simon. And I'm particularly looking forward to this one. They've been on stage with me at reInvent, so I'm pretty sure we've got a pretty good story to tell. Oh, it's a good one. It's a good one. And the two gentlemen who are joining us to tell the story, the first one is Morten Keldabach, who serves at the CTO of Too Good To Go, where he leads the technology strategy for the world's largest marketplace for surplus food.

0:47Morten, welcome to the podcast. Thank you, Simon. Robert is a key technology leader at Too Good To Go, having joined the company in early 2018 as well. Welcome to the podcast, Robert. Thank you, and it's good to be here. Let's get into the story here. Now, Werner, you had these two gentlemen on stage, but before they were at Too Good To Go, they were both at Endomondo. And so there's a story here, isn't there? Yeah, so can you tell me what you actually did before you did Good To Go? Yeah, maybe I can start here, right? I can say that back in 2009, I was still working at Nokia at the time, right?

1:30And GPS technology was an awesome new thing. We all knew these trackers from our cars, you know, but then it became popular to track your running exercise. And I just have to say that that was such an awesome idea. and I learned three Danish founders had started this company called Endomondo, which was all about kind of making people run and making people become healthy and do sports. And I felt that was just something that I wanted to be part of. It sounded so awesome because I like running, I like biking, I like staying healthy. And now this technology became kind of a lever for doing that. And yeah, so in 2009, I got an opportunity to join this, and it was a very early startup, right?

2:23There was one engineer, and the three founders, none of them were engineers. So I got the chance to join almost without salary. What a great opportunity. What a great opportunity, yeah. and back in 2009 the iPhone was here but it was not it was not very widely used so my first task was to build a Symbian sports tracker for Nokia phones so that was my way in and there was one other engineer at the time and he was like he was doing the back end So there was me and him, and we had a few students helping out as well. But basically, the two of us, we had to figure out everything. A mobile protocol, how does the mobile speak to the server?

3:24We had to design the protocol. We had to realize that in 2009, using HTTPS on mobile connections that went through all the mobile operators was actually not something that we could do. So we had to learn a lot of stuff. And eventually we figured it out and the company started growing. Eventually in 2012, we could hire a lot more people. And I think that's when Robert came on board. That's also when Skylink starts to happen, hi? I came from a different background. So also an engineer, but also coming from financial insurance. It's a very different beast than a startup. But I was like, hey, I want to try something new.

4:15You learn something. You want to go on something. And I actually started running a bit more, being more active after, let's say, a slightly sedated lifestyle for a couple of years. And I felt all this opportunity was kind of fit into it. and just joining a startup was, it was, it was fun. I mean, we were younger back then. A lot more time. A little more energy, a little more hair. Yeah, more energy. Yeah. And I stumbled upon this job post by chance and they were running a, I don't know, I don't think a lot of people run this production anymore, but Apache Wicket system and that has state, which we'll come maybe touch upon a bit later.

4:56scaling state on server side is, I can see, Werner, that is not something that's fun. It's not something you want to do. I hope you did get paid. Yeah, I did get paid. I think we had started to pay salary at that time. So both of you had some of those really interesting dive deep lessons early on, relatively early on. and then you've come together at Too Good to Go. Firstly, for all of us listening, what is Too Good to Go and what's the purpose of it? Too Good to Go is a company that was founded with the purpose of fighting food waste. It was originally founded back in 2016, maybe even early 2015, with the idea that kind of the original idea was a buffet restaurant.

5:57They were throwing out way too much food at the end of business. So the founders, they decided that, hey, some people would be willing to buy that food for a reduced price. And the restaurants would save money. They didn't have to throw out their food. They could even turn their potential waste into a bit of cash. And the planet would also benefit. So it was like a win-win-win, as we say in Too Good to Go. The consumers get a bit cheap food, the stores get a bit of cash, and the planet wins. And then they realized quickly that it was not only interesting for buffet restaurants. A lot of food businesses have this problem that they like to have food available even at the end of business.

6:49so just before they close down there's still a lot of food, fresh food available and instead of throwing it out they could, we gave them an opportunity to sell it. So that's what Tukutu goes about. Where does the food normally go to? It normally goes to the bin. Yeah? Which is it sounds crazy, right? It's a landfill. Yeah, it goes to, it's landfill, right? and then we have this way that it's a win for the stores they don't have to throw it out basically no one loses this is one of the few things that we can actually do without any there's no loss in comfort or anything everybody wins I always thought to do good you need to be an NGO a non-profit but it turns out that there are many cases where you can do good have a good business and your customers are severely helped by it as well so that you do good but you still make a good buck yourself yeah yeah and i think uh i mean this is one of the approaches we we we also have is our ceo menelugge talks about is like if if you are an ndo you are dependent on someone else making sure you can operate.

8:15You need other funding from government or from donations. And she says, and I think she's right, is a strong believer in that, hey, you can make a win-win-win. You can make a business out of it to ensure that you can continue to operate. Because if you, let's say, let's say we do this, we scale out, we can do great. But if we're not here tomorrow because we can't operate, we have no money, then what's the point? Yeah, especially with uncertainties like USAID and others where companies or organizations suddenly can no longer operate. Yeah, it's a different story. But congratulations for your model.

8:58Thanks. And I love that we're talking about waste not, want not when it comes to food at the same time as we're talking about the frugal architect. So let's dive in. You know when you're watching a movie and something blows up within the first five minutes. And that sort of sets the stage for the story. We're going to start with that moment because you had the classic IT challenge of a business or an organization doing really, really well. And it's actually a massive headache because you're doing so well. So it's a good thing, but a bad thing, but a good thing. What happened during the 2018 World Cup besides your team not winning?

9:35Tell us what took place and what that did in terms of your architecture and what it was and what it went to. So basically, soccer, like good old soccer, is a big thing in Denmark, right? And there was a world championship in 2018. And at the same time, I mean, our technology team in Too Good to Go wasn't very big at the time, right? So I think we can safely assume that everyone was watching the soccer game Denmark versus Belgium. Unfortunately, I think we lost, but that's not the point of this story, right? But the point is that without us knowing, at the same time, our French country manager, kind of our leader in France, she went on national TV in France and France was not playing this evening, right?

10:29So there were actually plenty of people who were watching about too good to go and we crashed and burned, right? We got so much attention in France at our servers, they broke down. I think I had been with the company for less than two months when it happened. So it was kind of like an eye-opener to me how unready, how not ready we were to handle such a situation. We crashed and burned. And if I remember correctly, it actually took us some hours to get back up. We basically had to wait until the traffic softened until we could get back up again. Thundering hurts, thundering hurts. Just bring it in.

11:14Yeah. And, yeah, I mean, it was a moment of pride, right? Because, hey, we are so popular, right, that our servers can be taken down. That's awesome. But it was also kind of a big headache to figure out how we would avoid the same situation again. So do we know why it happened? Oh, yes, I think that's on me. So I did join the company a bit before Morton. And so the initial server architecture was basically a PHP app running against a SQL database. There are some limitations that comes with that kind of architecture. You will rely heavily on the database. There's no state or caching naturally built into the language.

12:04It can be done, but it's not naturally flowing as other architectures. And also the single database. So the moment you just, even with auto scaling on, when you can scale compute, you can't really scale the database. And then suddenly everything just hammers down. And at that point, you kind of lose it. Even if you manage to scale compute, it will be stuck on database connections or services. and they will be taking out and rotated and you can't really do anything about it. So yes, we kind of knew. So the people who built it, they had a great business idea, but yeah. And you build what you can build at the time.

12:46This is, there's this, you know, premature optimization, premature scaling. This is the, this is why we need smart people because it's like, well, at what point do I make that decision? And unfortunately the business doesn't always let us know what they're doing and where they're doing it. But I guess, Werner, you've seen that many, many times in many, many organizations, that this is the unpredictability that the cloud is kind of there to help with. How did you get it back online? Yeah. Well, actually, as Morten says, we did wait, and then we came online. I think at that point, we were actually slowly going over to AWS, but not really using the infrastructure.

13:23So like a lift and shift. and if you do lift and shift for I think it was a cPanel server somewhere if it brings a bell with some people which is like a single instance everything running in once which is great unless we have scale so doing it's not enough to do a shift and lift we learned that during at Endomondo at you can say similar problems but also at scale you need to think you know you have a lot of opportunities even in 2018 now you have even more opportunities. There's like tons of things you can do. But to get it up, well, we have to wait for traffic to go down. And then, you know, what took us down, right?

14:04That's the next question. Is there any monitoring? Not super a lot, but it's very obvious we have when if you ever seen the app, we like a lot of apps, we have a listing the moment you open. And that was a very, very, very huge SQL query. So it will take you down. It's only a matter of when. So the first thing we did was, okay, we need to do something about this. What can we do effectively, like fairly fast? And the first was like, hey, let's just take the services that are the most heavy, that we can start moving over to another architecture, possibly some caching, that would be nice. And then just move endpoint by endpoint to that new architecture.

14:48So maybe for listings, it's like you don't need to think about what's your business transaction. You need to show your product. And you start thinking about eventually consistency and things that we may assume that everyone does. But if you just do a school project architecture side, it doesn't really fit into that. I mean, maybe to take it a step back, when this happened, we already had enough signal from the business side that, okay, we are actually working on something that is successful here. So we knew that we had to be able to scale it, and we had already started talking about what to do.

15:29And the team size at the time, the total tech and product team in the company at this time in 2018, well, I mean, it was in the neighborhood of 10 people. So you can do a lot with 10 people, but you can't do that much. right so we we had a lot of decisions to make right and one of the decisions we made was to we are going to build something in Java right kind of that was our first architecture decision and why it wasn't the latest and greatest but we knew it was reliable we had people on the team who understood and we knew that we could recruit people in the area of Copenhagen a lot of people have that skill so we kind of That was our first decision, right?

16:14Our second decision, I think, at the time was to use as many services as possible from Amazon, right? We were already logged in on Amazon, so that was also an early decision. Use as many services as we can, right? And build only what we have to build. And then kind of the third and very important decision that we made, because probably as many other companies in this situation, we had this discussion, Should we pause feature development and rebuild everything from scratch and then kind of keep the business on hold for like a year or half a year or something like that while waiting? And we made the decision not to do that.

16:55We are going to do this gradually. So we kind of slowly started kind of, I think first we did an experimental endpoint, which was not very heavy on traffic just to prove the concept. but after that we took kind of the most critical parts of the system and rebuilt them one at a time right so basically and i think we were super super optimistic at the early days thinking that we would be done in like half a year of course end of 18 right and in reality we we retired the last line of php code two years later in mid 2020 but it was okay no one cared because and it's hard you're sort of rebuilding the airplane while flying the airplane and in my experience when people are sort of changing languages the phrase how hard could it be always comes up and then two years later you find out plus the people who built the airplane aren't there anymore yeah exactly So we need removing a single line of PHP and nobody complains, nothing breaks.

18:13It's one of those cases, yeah. And this was tens of thousands of processes running and things going on. This is not a small system that was running. It was doing a lot of stuff. Well, you have to remember that at Amazon Retail, we used to use Perl as our language for page description. and you know at some moment you start to realize that there are no good Perl programmers so you need to move away from it I mean yeah write write only language don't read other people's code so so then let's let's let's take the next step and it's interesting because as you're telling this story to remind me there's a presentation we've had at reInvent from the solution architects almost from the start I think it's called scaling to your first 10 ,000 users that's 10 million users I should say.

19:05And it tells exactly the story that you're going through and then those steps you take. And it's interesting how folks have to, you have to go through those phases, but you had, you had one that I think is, is a lot of companies and software providers want to get to, but not all get to, which is suddenly you got to go international. And again, that's one of those amazing gifts of, wow, we're going global, but it's also, oh my goodness, we're going global. So give us some context of what was going on and how you tackled it. Yeah, I think so. As Maud mentioned in the mid-2020, we were like, okay, we retired the last piece of PSP Co.

19:41Great effort, guys. We're also launching features. So it went well. I will call it a success. We were just optimistic about the timeline. but then it's like Nest said hey we're going to go global we're going to go into the US and they're like we were like okay US was what does that mean for us well that means latency we were in the European data center in Ireland which I guess a lot of European customers start off in and the latency tests were a bit poor from our mobile experience so what do you do you have a you have an architecture you have servers they're not built for multi-region So how do you scale that?

20:21That was kind of like, what is the experience we want to give to our... We want the users, both our consumers and partners, to have a first-rate experience in the US. And how do we take that challenge? So we ended up with a solution where the application itself didn't really need to know a whole lot about being in multiple regions, but we would simply root. So you have a home region, and then there is a single component that knows how to root or fetch data, no matter where it is. So you do want customers to migrate between your European good-to-go and the US one. It's not that you have a clear separation between them.

21:06No. So it is. And then, of course, when you build for two, we also build for multiple things and also expand it to Australia later on. so it's basically that you can go as a customer in the US if you just fetch the feed where the data is in the US it will just stay there but if you have to interact with some data that's yours we'll either have to fetch that relevant data or root the entire call to your home region and that is built into it and something that we have standardized so each developer just needs to remember they need to root but otherwise don't really think about it so the customers are worldwide but I assume your offerings are extremely local so basically we don't now we do a little bit but back then we did not offer delivery so the food offerings are super local because the consumers they actually have to pick them up at their local store so they have to walk to the store so typically you will buy something that is within a few kilometers So it's from where you live or where you work or something like that.

22:16But some of the thinking behind not running completely separated architectures was rooted in business, right? During our expansion in Europe, every time we open a new market, a new country, we have a kind of a cold start problem, right? We need to get some stores onto the platform and we need to get consumers. and we learned that the tourist use case is actually not nothing, right? We got a lot of, in the early days of every new market, we get a lot of help from people traveling, right? So we did not want to miss out on that one, right? We wanted tourists visiting the US from Europe to help us drive the initial growth and so on, right?

23:04So there was a business reason behind it. And we have not regretted that one. From the vendors or the restaurants perspective, do you have to go out and find them? Or is this something where they come to you? Yeah, we have sales teams in every market where we operate. Okay. And typically, we start from no awareness at all. that is why and and then kind of we have the outgoing sales teams and they reach out and then eventually when too good to go becomes more famous in a market then we start having kind of the inbound requests and then kind of that also kind of helps helps the growth machine so very cool excellent and so so one of the things you've spoken about in terms of this architecture and some of the choices you made was boring, quote-unquote, technology.

24:08And not in the pejorative sense, but help us define boring for us and why you like boring. Yeah. So actually, we like to be boring, but not stale. Good one. That's good. I like that. So let's say in the case of Too Good to Go, right, you have some people, there was young entrepreneurs, they're not technologists, but they got a great idea and got a really good start. And like, if you then come in and say, hey, we're going to change it out, I'm going to use the latest and greatest, then I'm actually betting that business. I'm taking that bet. And, you know, I don't know how that's going to go. They don't really care about technology in that sense.

24:51Like we as technologists, we care about technologies, but I also like something true and tried in that sense that it's a safe, right? You need to know how to use it. It's not enough that, hey, now we have the newest language or if it's a JavaScript, you have a newest library number 1 ,000 this month and you're going to change everything and someone deprecates it. You have to be a bit boring. You have to, you know, you're going to live with these choices. Software, like I was in an insurance company, there was mainframes around, there was PL1. and for those who doesn't know what PL1 stands for, it's Programming Language 1.

25:28And this was in the 2000s. So it's like your choices will, you have to plan for long-term. So don't bet the business just because you want to try something. Be a bit safe, but also make sure that you continue on because technology to progress. And as there's, I forgot who the famous quote is, like technology is progressing at a faster pace than it used. Never had. It's always going faster and faster. It just gets faster. Make the smart choices and then maybe if you can avoid really over committing to something, then you have a choice forward. So that's what I mean with, we mean with boring, you know.

26:13Choose something that will work. Are most of your developers still in Denmark? Yes, we have the bulk part of our development team in Copenhagen on side and then we have a reasonable size team in Paris and then a little bit in Madrid, right? But we are still trying to grow out of Copenhagen. I think we are around just below 200 people now. So, and as Too Good to Go has become more and more famous, then we are still able to attract good people in the company. But I mean, eventually, we will run out of talent in Copenhagen and we will have to expand further. And it's interesting too, I think as you're growing, you're thinking about doing this again frugally.

27:05And just to give a sense of scale, now this is a platform that has over 81 million registered users worldwide, 145 ,000 active business partners. So there's a lot going on on the system. And one of the philosophies you have is always ask why. Help us unpack that because it's interesting as you're growing and scaling and clearly moving quickly, different countries, it's kind of like, well, we're doing stuff. Always ask why sounds like an interesting sort of pause for a moment. So what we want to do is like when you do something, it's like don't do it just for the sake of doing it. It's like get down to the call.

27:46Why are you doing this? Why are you implementing this feature? why are you taking this technology choice? Like, are you doing it just because you think it's fun or because it's going to help the business or it's going to help us scale? Just like there needs to be a reason and it needs to be a good one because every time you take a choice, you also decide not to do something. And we do have a tendency, we are, as humans, biased for action. But sometimes you do like, maybe you shouldn't or maybe you should just cut down. It's very easy to get excited by. I can also get excited by technology, but I tend to like, okay, then I'm going to play around with my laptop rather than saying, let's change the entire backend infrastructure.

28:31So it's about why are you doing something? It's about that purposefulness, I think, comes into it. That's a challenge. I know a lot of folks really struggle with it because like I say, things are moving so quickly. Like, you know, everyone's talking about LMS and Gen AI, and there's this new language and that new language and this framework and this, like, and what you're suggesting here is it's important to just sort of like take a beat. So if you take the - Be on the ball for the right thing. Yeah. So if you take the LMS and AI, it's like, you're also like, what problem are you trying to solve?

29:05Right? If someone goes, goes tell you, I'm not saying that there's probably, there's definitely problems you can solve with AI. It's like, hey, let's use AI. and like, okay, what problem do you want to solve? Because there's also a thousand choices, just ALMMs, there's like exponential choices. So what is the problem you want? And then we can find out, like we don't need to maybe try every single ALMM to try something, but like what's the core of the problem? Where, always when I have a problem, like how do you see, hey, you come to me with a problem and my first question would probably be like, so once we have solved this problem, how does your world look?

29:44What's going to be different? What does your workflow do? Or what are we going to do differently before we start even thinking about building anything? And also, a lot of the times, you never get to the full end. You get, well, enough that you get the value, and then you continue. I do notion that most of my customers are overwhelmed. I mean, there's 10 new models even read. And so there is so many stories continuously that it's a hard time for customers to make a choice. Do what we need to do. Do we really need to? Then I really like your approach. What's the problem that we try to solve with this?

30:30Yeah. So I think you guys are on a very, very well path. many of my customers would like to push a pause button yeah we're living in exponential times um i mean it's a hard time tracking it yeah but i think robert and i we we both also we've we feel a lot of pressure to kind of just as anyone else i assume that right in these ai times here we we are also scared of like falling behind right so so it is like i i totally get that fear that that is all over the place. But I think we are taking an approach where we experiment also in areas where we see, okay, this has something to it. So we are experimenting.