Modern Distributed Applications with Stephan Ewen

0:00A major challenge with creating distributed applications is achieving resilience, reliability, and fault tolerance. It can take considerable engineering time to address non-functional concerns like retries, state synchronization, and distributed coordination. Event-driven models aim to simplify these issues, but often introduce new difficulties in debugging and operations. Stefan Yuen is the founder at Restate, which aims to simplify modern distributed applications. He is also the co-creator of Apache Flink, which is an open-source framework for unified stream processing and batch processing. Stefan joins the show with Sean Falconer to talk about distributed applications and his work with Restate.

0:44This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:02Stefan, welcome to the show. Thanks for having me. Hi, Sean. Yeah, absolutely. Thanks for doing this. I'm excited to get into it. So I wanted to start off with a bit of your background. What was your journey and experience from working on Flink to now being the CEO and founder of Restate? Yeah, most of my professional life was Apache Flink so far. As part of the team that started it in 2014, and in a way, probably I'm responsible for a lot of the early architecture of Apache Flink around the way the data plane, the coordination, the snapshots, and all of that worked. The journey actually started even earlier.

1:42In a way, it started when I was still at university and grad school, and we're working in this sort of intersection between Hadoop and databases and some of the very, very early steps of stream processing had just come up, you know, like Storm was a new thing back then. And after that, we actually took the project that we worked on at university and turned it into an open source project. It was sort of a mix of a pipeline batch processing system and maybe some early steps of a streaming system. And like as part of the open source journey, we found our sort of sweet spot with users and stream processing, turned it into a stream processor.

2:14And then, you know, I kept writing that wave of Kafka, Flink, the sort of advent of real-time stream processing, stateful stream processing, unified version stream processing, and so on. I left that space roughly 2021 to focus on something new and sort of a project that, or a set of problems that caught my eye back then was kind of similar to the problems that we're trying to address with Flink. Flink being sort of an analytical system for robust analytical real-time pipelines. And we're more and more being asked for how do you build more like transactional event-driven application, not the applications that aggregate events and join events and so on.

2:52And yeah, feed dashboards, feed recommenders and so on. But the type of pipelines that in the end actually process payments, invoicing, orchestrate orders and shipments and so on. Like these types of applications that folks are sort of stitching together manually with databases and queues and lots of custom logic. And it felt like folks were looking for a solution. They even turned to systems like Flink to kind of implement that. It's not a great match. You don't use an analytical system for transactional processing as a general rule of thumb. But I guess this question came up more and more. We thought, okay, we should probably start looking into that space and building something.

3:27And that is when we started working on Restate. We've been doing this for around about two years now. And there we are. So how do you sort of describe Restate? Do you term it or do you sort of bucket it into this class of like durable execution frameworks? Yeah, it's definitely puts durable execution as one of the main ingredients on its list. So you're right, there's this big bucket of durable execution engines. It almost seems like there's a Cambrian explosion of those right now, like there's a new one every few months. And Reset is definitely a good candidate if you're looking for a durable execution engine.

4:02It's a little more than that, though. It's really, I would say, a more holistic platform for building distributed resilient application. It doesn't just include durable execution as in being able to sort of journal different steps in your process and being able to reliably recover them, which is this notion of, you know, like workflow style logic, but implemented in general code and general purpose code, not in like DSLs or so. So Restate goes quite a bit beyond that. Restate sort of tackles the more holistic problem of like, what if we try to apply this idea of durable execution, not just to a single workflow, but what if we sort of incorporate concepts like distributed communication, a state that outlives an individual like workflow or an individual durable execution?

4:45How would all those things interact? How do you build sort of a more general platform that applies that level of durability and resilience to distributed services in general and not just like an individual workflow? Going back to the sort of rise of durable execution frameworks and the idea that there's sort of like, seems like there's a new one every six months or whatever. What do you think that is the case? Like, this is something that people are investing time into, and it seems like there's growing interest in it. Yeah, I think this is because the state of the art is sort of, it's not feasible.

5:18I think it's more and more developers and companies reaching that conclusion that the challenges that you're facing today when you're implementing distributed apps, it's just not something that many software development teams can handle. And the ones that can handle that, they're not really using their time well, because they're spending most of their time really on problems that have nothing to do with the business logic. They're spending their time on problems like, you know, figuring out race conditions and how to avoid split brains and how to avoid lost updates if a zombie process appears and all those things, you know, they should be focusing no one making, you know, adding features to the application and not creating the workarounds around distributed systems problems.

5:58I think this has gotten particularly bad with the rise of microservices. And I think it's a big part of why there's a little bit of a backlash even against microservices right now, like for all the benefit they give you, I think many people realize how challenging distributed infrastructures with lots of microservices are. And, you know, some are just saying, okay, let's go back to the monolith. It was just a bad idea in the first place. And then there's a whole group of people that say like, no, no, we actually like a lot of the benefits that microservices give us. We just want like a stronger foundation to build them on.

6:27We want something that sort of frees us from dealing with many of these problems. And I think this is where the whole wave of durable execution systems got started, kind of that movement. And I would say it gets actually more and more necessary to have those systems because applications get increasingly distributed, right? And it's not only the services you built yourself, but more and more functionality that you access is hidden behind APIs provided by SaaS vendors and so on. These are all services that you interact with. They become part of your microservice architecture, even if you don't really own them and so on.

6:59But they add to the complexity of the problem. And that's a trend that's only increasing. I think that's not going back. Yeah. Even though he said, like, I hate microservices and down with microservices, I'm going to go back to the monolith. Even if I build that monolith and I deploy it and I'm able to manage that, I doubt that application exists in isolation. It's going to have interdependencies to third-party services, which then are going to reintroduce essentially all these distributed system problems. So if I'm connecting up microservices or even I'm calling like a third-party API, then there's all kinds of things in a distributed system that can go wrong.

7:33There can be certain outages. So without using some sort of framework to help me solve those problems, what are teams typically doing to do that? Are they just sort of making those requests and then at best they're doing some sort of retry scheme with exponential back off to see if they can push that through and they're okay with sometimes that not happening? Or what is it that, you know, companies are doing to try to solve these problems now? Yeah, I think there's a lot of different approaches to that problem. There's, I think the first observation I would throw in is like, there's a lot of companies that actually don't really get it right.

8:06You know, just the fact that there's still so many websites that tell you like, don't hit F5 while you're undergoing a booking on order process is like one indication that they can't really handle these like things well, like concurrent requests. You still see lots of like a lot of artifacts that if you are a developer, you can, you something has gone wrong in their backend right now. I would say just like, first of all, a lot of times it's not actually getting solved correctly. And I actually heard a quote in another podcast from somebody who works at a food delivery startup who said like for many, many years, their solution was just like ignored and sent a voucher until it like at some point just became really expensive.

8:39Ignore the problems and send vouchers. I would say if you want to actually solve the problem, one of the ways to do it typically is to stitch together different systems. The typical ingredients will be use a queue, a database, build your own retry loops with backoff. But it's not just as simple as implementing a few retries with backoff, right? Like you have to always kind of worry about, okay, what actually, if I guess the retry actually happening, if you're triggering this as an RPC call, you know, you might be trying it, the whole process might actually go away. Then there's a next step, you probably put a queue in front, actually say, you know, even if the process gets away, the event gets redelivered somewhere else.

9:14Now you have to actually worry about the fact, you know, you might actually have two processes that work on the same event twice. Like, are they overriding each other with their retries? Do you want to throw in a lock? Do you want to introduce versioning and conditional updates and so on? Then, you know, you might be interacting with APIs. You might call them, get a result, crash afterwards, recall them, might get a different result the second time you call that. So the next retry actually follows a different control flow than the first one. And things go completely haywire. It doesn't stop with a retry.

9:39I would say it's very often you start with a queue and a retry, and then you incrementally just add like bits to guard against that bucket that you discovered at that. And then incrementally, it just grows really complex. And it makes a hidden assumption on this is exactly how that queue behaves. And that's exactly how that API behaves. And then somebody changes that and everything breaks again. And then you're back to fixing this. I don't know. I don't think there are really good solutions. If you go to the extreme end of saying, okay, here's an extremely sensitive, high value process, then sometimes folks throw in workflow engines as one solution, right?

10:08Let's say here's an order process that we really don't want to go wrong because that can actually cost us a lot of money. you might pull in a heavyweight workflow orchestrator, but it's really not something you typically pull in for like small microservice logic, first of all, because it's really a complicated component to have in the stack. And second of all, it's sort of really foreign citizen. It doesn't interact well with a lot of the other logic. Yeah. And it's probably a little heavyweight for the majority of the types of calls that you might be making in between microservices or even to external services.

10:40Yeah. And this is actually one of interesting things that durable execution brings in specifically in like an implementation like we're looking at at restate if you can actually make durable execution cheap cheap as in low latency having a durable step introduces a very moderate latency overhead then what you can actually do is you can start assuming these workflow style guarantees for a lot of code in your application it's no longer sort of prohibitive to do that from two sides like it's no longer so slow and expensive that you say oh i really don't want this here this is in the synchronous path of the user interactions is going to make everything really sluggish, but it's going to feel still fast doing that.

11:15And the second thing is you're still writing code. It still fits in with all your tools and with all your deployment and pipelines and all your versioning, all your schema registries. You can still keep using that. So it feels like you can keep doing mostly what you are. You're just adding this fine-grained reliability to your functions and get a lot of the problems out of the door. That's actually the ultimate goal of systems library state. Okay. Going back to 2021, when you started working on this, where did that project start? How did you even begin to try to tackle this problem? Yeah. So we started initially actually trying to solve this from the side of Apache Flink.

11:54And as I mentioned, we were working with a bunch of users on analytical pipelines. And then this question came up, users building sort of transactionally meant driven pipelines on Flink and we really didn't find that a good match and, you know, very moderate success. There were a few that could make it work with like specific approaches, but not generally good experience. And then we started a subproject in Flink. It's kind of still around. It's called Stateful Functions. The idea was just like, let's do the thing that folks really like, which is just reliable communication, transactional state, and sort of encapsulate that into an individual piece, an individual function.

12:26Think of it as like a Lambda function, but when it's invoked, it has contextual state. It's invoked in the concept of a key, sort of attached to a key. And when it's invoked, it's sort of hydrated with the state of that key, can interact with that, modify that. It can basically produce a set of sort of RPC or messages that go out to other functions. And then this is all like transactionally committed, like the messages are sent to other functions, the state is committed. It's almost like a stateful, disaggregated, serverless actor system. I think of it like that. So that, in principle, raised a lot of interest.

12:55Like there were a lot of folks that did like that as an abstraction could see there's like, this is great for building anything from digital twins to, well, yeah, transactional state machines, which represent orders, even invoicing payments, and so on. Fox did actually build payment processors on that. We learned that only years later. Fox was quite crazy. There's just like one linchpin that this had. And this is, it was built on Flink as an analytical system. And Flink's really throughput optimized. It's like low latency for an analytical system, but that is mostly if you sort of use the at least once semantics in the things, meaning you sort of like you push events right when they come.

13:33If you want actually transactional results, you're introducing a huge latency, namely Flink is checkpoint based, and you have to wait until the next checkpoint happens. That's sort of the latency you introduce if you want to say, I don't want to take a second step before the first step is really durable. And that is in the seconds, right? And imagine using this as a foundation for workflow style logic. It means like every workflow step is sort of a, let's say 10 second latency. It was like completely impossible to do that. And it is also something that the EFLink architecture could never like fully remedy.

14:01So we thought if we really want to make that happen, if we want to make that vision happen of durable execution being something that's so low latency that you can use it without worrying about introducing latency overhead, even in sort of latency critical paths that are like synchronous interaction paths. So then we'd really have to build a new stack. We'd have to start from the bottom building on a low latency log, on a low latency architecture that emphasizes fast durability and not analytical throughput. And that's how we then got started with Restate. So what are the core building blocks of Restate?

14:32Both from a user perspective, like what am I sort of stitching together from the developer experience? And then what is sort of the architecture behind the scenes that's helping me essentially support that in a way that is going to shield me against these kind of like outages or other issues that you might run into in distributed systems? So yeah, there are very different levels from which to look at. Let's look at it first from the sort of infrastructure side. What does Reset actually sit in your infrastructure? You can think of it, it takes a similar place as a message queue or a message broker.

15:00It's kind of a, or a workflow orchestrator. It's kind of a marriage of, let's say, the Kafka-esque event-driven application and sort of the temporal-esque durable execution workflow world. So it sits where a broker would sit. You're writing your logic as service handlers. The abstraction is, we try to keep microservices really as the abstraction. So you're writing services like almost as if it would be like a Spring Boot application or Express.js application or so, so like handlers grouped into services. And then restate is the queue through which those services sort of get triggered, right? So if you want to actually trigger a handler, you put an event in that queue that's supposed to invoke the handler and then restate invokes that service.

15:39So in that sense, sort of classical queue in front of the service. The abstraction that we exposed, though, is not really that of an event, but it's more that restates or looks at the services and their handlers, re-exports them and sort of becomes a reverse proxy. So we're really trying to get away from people thinking in terms of like queues and events and trying to keep thinking in terms of like synchronous and asynchronous RPC. So that's really how you build it. Think of it. It sits in the infrastructure like a Kafka services, but it exposes itself as a reverse proxy that sits in front of the services.

16:11So is the main advantage in sort of comparing this to doing something like event-driven architecture with Kafka, like is the main advantage just having like sort of a level of distraction from having to think about events and queues and so forth as the person doing the implementation? If I'm doing the implementation, I can kind of just focus on the work that I need to do and that stuff sort of distracted away by restate. I guess that's one way to think about it. I think if we go in the details of what the programming model has, maybe we'll see this. But in general, you can think of it as it's like it's a level up from sort of Kafka style event-driven application.

16:46You're not thinking in terms of queues and events. You're thinking in terms of durable, stateful, resilient invocations or functions. Yeah. And that sounds like maybe an academic detail, but it actually is a word of a difference because that actually means that Restate takes on a lot more responsibilities. It doesn't just take on the responsibility of say, okay, I'll deliver the event and make sure the function is triggered. and it's, you know, it's redelivered, it's retriggered on a failure. It also understands, okay, how do I fence retries against, you know, earlier executions? How do I log contextual state, attach contextual state?

17:20How do I track progress if I have like multiple steps that happen as part of a function invocation? And I want to actually understand that I record the result of the previous step before I start the next one, just, you know, to give ourselves an easier life when it comes to implementing, you know, complex control flow that would be thrown up if it's on different results during different retries. All those things, if you implement them manually, you're typically not just looking at a queue at Kafka, you're typically looking at combining a queue with a locking service, with a database, with a scheduler, and so on.

17:48And so Restate wraps that all together and says, we're going from queue and event to doable stateful, resilient function executions. And then, as I mentioned before, the core sort of programming model is services that are meant to mimic RPC style service frameworks. And the sort of the simplest building block you have is really service handlers that get durable execution and then resets of layers, a few things on top of that. Like one concept is virtual objects that are stateful handlers that remember state across individual invocations, shard around keys, and then, you know, more like high level workflow constructs where you can actually add signal handlers and query handlers and so on.

18:29But all of that is sort of built on top of the general service abstraction. So if I'm implementing one of these handlers and it gets executed, what is sort of the life of that process behind the scenes? Let's assume you're implementing a payment processing handler or something like this, and that gets invoked. And let's say the logic that you have in there is, I first have to check the status. Like, was that already processed? Was it maybe canceled? Was it blocked before? So let's say the payment's identified by an ID. I might want to call a fraud detector. I might want to update a database, send them out a message, and so on.

19:02The lifecycle of executing this would be the following. Some external trigger, some external client says I want to execute that function. That enters Restate, the Restate server, the broker component, as an event. And Restate will understand, okay, where does that service live? You can think of it that the service has to be registered at Restate, the service endpoint. You know, where is that deployed? Is that like here, URL on Lambda? Is that an HTTP2 server endpoint? Is Kubernetes deployment and so on? You have to register that at the server and then the server connects and pushes the invocation.

19:33You've worked with, for example, like Amazon EventBridge or things like that. It's kind of a very similar model. So Reset will then look up, okay, that, you know, that handler is on that endpoint and I'm connecting to that. Let's assume it's an endpoint on Kubernetes or so. In this case, it would open a streaming connection, HTTP2, put the invocation and then hold on to the connection. and that is sort of the lifeline to that single invocation or execution attempt, which allows the service to stream back things like general progress, state update, outgoing messages. The function, let's say, or payment handler would also get when it's invoked.

20:07You know, if it's a stateful handler, a virtual object, ReState would attach all the contextual data it knows for that individual handler to the invocation so that the handler could directly look up things like, okay, what's the status that the previous status that was committed, like, okay, it's still new. So let's start and execute that payment. And then let's say we're calling things like the, let's say we'll call an external fraud detector API, we get the result, and we say, okay, this is actually a durable step. Then the handler would put the result of that step into that stream that goes back to restate.

20:37Restate internally has a consensus log that persists all the things it receives. And it has a bunch of sort of logic around this to understand, okay, is that information that still comes from, you know, like a valid execution attempt? Or does that come from an attempt that has been fenced off in the past? Does that, you know, it has like a sort of, yeah, it has sort of an elaborate consensus log that supports a conditional append of that operation to the journal. And it links that operation or that entry, that's the result of calling the fraud detector API onto the original event. And if we'd say, okay, there's a failure after that point now, that failure could be, you know, like just the connection is ruptured, the process goes away, or there's a timeout, then the reset server would understand, okay, that event hasn't been completed at the execution.

21:24I didn't actually get an acknowledgement back for that yet. And it would send the event to another process. It would retry sending it to that endpoint, and it would attach everything it has to that event. And that's the contextual state like last time. But now also it would attach things like the journal entries that it already collected, like here's the result from the previous step, wraps that all up and sends it there. Then let's that service basically say, as I'm going through the code again, I can skip over steps that I've already completed. This is what the SDK library basically does for you, like understands, okay, that's already found in the journal.

21:58We can ignore this. This is a new step. We actually add an action or an event for that in the journal. And then it goes on. That applies to pretty much any operation. Recording the result of an API call, updating state, sending out a message, all these things basically become events that are streamed to the reset server. And the research server understands how to process these events. They all get sort of attached to the original invocation, but sometimes they also represent more. Like they present an outgoing event that is then routed to another service, or they represent a state update, which is applied to an internal state index, and so on.