Federated learning in production (part 2)

0:03Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at Fly.io. Launch your AI apps in five minutes or less. Learn how at Fly.io.

0:44Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard and I'm joined as always by my co-host Chris Benson who is a principal AI research engineer at Lockheed Martin. How you doing Chris? Doing great today Daniel. It's a beautiful spring day here in Atlanta, Georgia and I got to say the flowers are coming out. It's a Yes. Nice day to talk. They're probably distributed all over the various lawns. Everywhere. Federated even. Yes, yes. Well, Chris, this reminds me. Last week, we had a kind of part one intro to federated learning and some details about that with Patrick from Intel.

1:33He mentioned recently that he was at the Flower Labs conference and the Flower framework around federated learning. He mentioned quite a few times. Well, we're privileged to kind of carry on the conversation around federated learning into a kind of part two on the subject, because we've got Chong Shen with us, who is a research engineer at Flower Labs. Welcome, Chong. How are you doing? Hi, I'm doing very well. Thanks for having me.

2:30episode 160 titled Friendly Federated Learning. It took me a second to say that one, but I'm sure a lot has changed and updated and advanced in that time, of course. Maybe just to start things out, Chong, could you give us a little bit of a context of your background and how you got kind of introduced to this idea of federated learning and eventually ended up kind of working working with Flower? Yeah, absolutely. Well, thanks again for having me. So my background is in computational physics. So I spent many years working, doing research in the computational physics field, both my PhD and a postdoc.

3:14So I worked a lot on parallel computing, on supercompeting classes. I was also very interested in machine learning and deep learning in general. So when I pivoted away from academia to go into what they call industry, there was this space where you have distributed learning. So that was in 2021. So when I started my career back then, it started as a sort of a data science consulting business, but specializing in federated learning. And I saw lots of projects that were very interested to adopt federated learning or this distributed learning approach to solve some specific problems that they have.

3:55But I also came across the Flower Framework, and open source development is a big passion of mine. So being able to develop a framework that is used effectively with a very permissible license, I think it's a pretty cool thing to do. So that's why I decided to join Flower Labs and become a core contributor to the framework itself. Yeah. Yeah. And I feel already connected with you because my background is in physics as well. It's always good to have other physicists on the show that have somehow migrated into the AI world. I'm wondering, in that transition, you mentioned this transition kind of academic to industry.

4:48You were getting into even consulting around federated learning. Was that idea of federation or distributed computing or however you thought about that, was that kind of a key piece of what you were doing in academia, which led you kind of into that interest? Or was it something else that kind of sparked the desire to really dig in there as you were kind of going into, to quote, industry, as you mentioned? Yeah, it wasn't something I came across in academia, surprisingly. But somehow, when I stepped into the data science world, I came across people who are looking into it. And that became an approach that back then we sort of adopted to try and solve some problems.

5:34So we saw that federated learning could be a way to solve it. And then it was very coincidental. Okay, it's distributed learning. It's distributed computing. So it resonated with me quite strongly. Yeah. And was that related to working with sensitive data or in regulated industries or something like in those consulting projects or just interested in kind of that progression? Yeah, actually, there are, I would say, two broad categories. One where the data is incredibly sensitive and we usually refer to them as really siloed data, data that should not absolutely leave the boundaries of where it was generated.

6:18And then the second group or second cluster is the problems where the data sources are so massive. The point at which the data is generated, generates so much data every second of the day that they just can't do any useful or meaningful analysis on this kind of raw data. And they have to do a lot of downsampling. So they try to look into pushing competition to the edge and try to see if they could apply some sort of machine learning approach or deep learning approaches on this sort of massively generated data without needing to downsample them. That makes sense. And yeah, I guess I should explicitly mention as well that in the kind of part one of this two-parter with Patrick, Patrick did provide a kind of detailed introduction to the idea of federated learning.

7:08And we discussed that at length. So if people want to go back to the previous episode and listen through that, that may provide some context. But it probably would be worthwhile just to give your sort of 30-second or couple-minute view on federated learning and how you would describe it at a high level. And then maybe we can jump into some other things. Sure, absolutely. The easiest way to think about it is looking at your classical machine learning approach. Classically, you need to bring all the data into a single location. think of a database or on disk, and then you train your model on that data.

7:52But sometimes it's not so easy to actually bring all the data into one location just because of the privacy reasons about moving your data, some geopolitical considerations surrounding it, and also the data volume that's been generated. So instead of bringing all the data to one spot and training a machine learning model on that, What you do is you move the machine learning models to the point at which the data is generated, and then you train these local machine learning models at these sources. Then instead of moving the data across, you move the model weights to a central server, which is much, much smaller.

8:34You can then aggregate the model weights to learn from these various data sources. And then over time, as you repeat many, many rounds of this, you end up with a globally aggregated model that has learned from this variety of data sources without needing to move data sources across. That's the essence of federated learning. I'm curious, as you guys have worked on the framework and you have new users coming into it, what usually prompts a typical user from your perspective to move into federated learning? Before they're really fully into it and they understand the benefits and they're sold on it, if you will, what's usually, in your experience, the impetus that gets them into that mindset and drives them in that direction initially?

9:24What causes the change in the way they're thinking? So they go, I definitely need to get into federated learning and go use Flower specifically. Yeah, absolutely. I think from my experience, the biggest driver is when they realize they can't move their data, right? But when they speak to all the parties involved, they say, oh, I have this data set. Oh, you have this data set. But I don't really want to share them. And then, okay, this is where federated learning or FL comes to the picture and decided, okay, we really need to do this. This is one aspect of it. And the other aspect is when there's this big company who has, you know, let's just say many, many data sources.

10:01They say like, okay, it's super difficult to coordinate all our databases together so that we can have a cohesive way to train the machine learning model. And this is also when you try to look for all distributed machine learning systems, then they realize, oh, they come across fairly learning. So there's these two vectors that drive the typical use cases. Curious if I can follow up on that, because I have a personal curiosity. I happen to work for one of these big companies that has data in lots of different places. And in addition to that, and we kind of in the previous, last week when we were talking, we talked a little bit about some of the privacy issues as well.

10:39I'm curious what you think about this. Like in our case, and we're not the only one, lots of that data is stored at different levels of security and privacy. There are different enclaves, if you will, where you're trying to do that. And how does that ramp up the challenge of federated learning when you have different security concerns around the different data enclaves that you're trying to bring together through federated learning? How does one go, instead of saying all different locations for distributed data are equal, when you're dealing with different security concerns, do you have any ways of starting to think about that?

11:18Because as I come into it as a newbie on this, that seems like quite a challenge for me. Do you have any advice or guidance on how to think about it? Yeah, yeah. I think this is from my experience is that the complexity of the solution scales with a number of data stakeholders involved. And when you mentioned about different levels of the enclaves, that to me sort of signals that there are many data owners who manage their data a bit differently. So the primary, the key to solving that is to harmonize the data standards first, to be able to get on to a federation. And then from then onward, the implementation becomes much, much easier.

12:10I think it's one of the key things that I've seen. And we've kind of talked about your background, the sort of introduction to federated learning, some of those motivations. organizations um maybe you know before we get into flower specifically and some of the more production use you know from your perspective as kind of being a central you know you're at a kind of central uh place within the ecosystem of federated learning i guess what at a you know just very honestly sort of because we had that last episode in 2021 from 2021 till now like what how is the state of adoption of federated learning kind of in industry different maybe now than before um how how has that grown or how how has that matured as a as a kind of ecosystem i guess yeah it's a very um yeah it's a very good question if i want to if i were to put a number to it i and this is really arbitrary i think there's a 100x difference from um you know 2021 when the the flower framework sort of existed and now.

13:24And one of the key changes in the usage of heritage learning is the ability to train foundational models and large language models. And this has been a significant change and driving force. So previously when we talked about using the flower framework, you may be confined to models that are not super large, small by today's standards of the order of millions of model parameters. But these days, when we're talking about making use of text data, image data for these foundational models, you are thinking about models at an order of billions of parameters. And there is a fundamental change in also how we have structured the architecture of our framework and also to increase the ability to stream large model weights.

14:20So all of these things are happening right now as we speak, and there's some exciting new progress. Hopefully, we release a new version in a couple of weeks. And for the users, the usage is identical. Nothing has changed. But what has been unlocked is the ability to then train very large models. So all of these really increases the appeal of using federated learning or the flower framework for a larger variety of use cases.

15:02you know what's beautiful about good code it just works no fuss no five-hour debugging sessions at 2 a.m that's exactly what nordlayer brings to business security while you're busy shipping features and fixing bugs, NordLayer handles your network security in the background. It's like having a senior DevOps engineer who never sleeps, never takes vacation, and never accidentally deletes the production database. Zero Trust Architecture? Check. VPN that doesn't make your team want to work from the coffee shop instead? Double check. Deploy in under 10 minutes with no hardware to rack and stack? Triple check.

15:40Built on the same foundation as NordVPN, but designed for teams who need granular access control and compliance reporting because apparently it works on my machine is not sufficient for the auditors. The good news is our friends get up to 22 % off plans plus an additional 10 % with the code practically 10. That's practically dash 10. That's less than your monthly GitHub copilot subscription, but infinitely more useful when the security team comes knocking. Check it out at nordlayer.com slash practical AI. Again, NordLayer.com slash practical AI.

16:20Well, Chung, as we've kind of dived into the show and we've already started making reference to flower quite a bit, but we haven't actually really described specifically what flower is in detail as a framework and what it brings and such as that. Could you take a moment, and we probably should have done this before, but maybe kind of express exactly what Flower is, what the components are, and how it kind of helps the user begin to federate their data in terms of what their workflow is? Could you talk a little about kind of the basics of it? Yeah, absolutely. So the Flower Framework is our flagship open source code that's built on the Apache 2.0 license.

17:11And this framework allows users, any data practitioners to build a federated learning solution. So with the framework, what this means is they are able to, I guess in code terms, install a basic Python distribution of flower and to build the different apps that allows you to construct the fundamental fairytale architecture. So what this means is to be able to spin up your server, which aggregates the model parameters, and to write the code to also do your training on the clients. the structure that we provide within the framework allows users to follow the same reproducible way to perform their federated learning so i think at the essence this is this is what it is what i also wanted to say that and one of the appeals of flower for me personally is that we really emphasize the user experience this is why we always say flower is the friendly federated learning framework.

18:15We want, we prioritize the experience of all our users. We support them on Slack. We also have a discourse channel called Flower Discuss where we actively answer any of the questions from users. And we also have a fantastic community that has contributed a lot of code improvements to the core framework as well. So we are completely open. And we're built transparently and really accountable for every single line of code that we commit to it, you know, to the highest standards. Yeah. And I can testify personally. We at Prediction Guard, we work with a number of students over time at Purdue University.

18:59They have like capstone projects. We're in the same town. So it's natural that we would, you know, work with some of those students. We've done that a couple of times now. And one of those student groups that we had, I believe it was last year, actually did this sort of capstone project related to federated learning and training language models, translation models, and trying various things. And they evaluated a bunch of different things, but I think ended up using Flower for the reasons that you mentioned. So they were newbies into this world of federated learning. obviously very smart students, no doubt there, but they definitely gravitated to the user experience with Flower because they had programmed in Python and it just sort of came naturally to them.

19:56So yeah, I'm sure that's a common experience that maybe you all hear from others, that sort of natural Pythonic way to kind of approach these topics? Yeah, yeah, we do. Absolutely. I'm very happy that you shared that experience. This is good to always hear feedback from your community. But yes, Python being the really driving language behind machine learning models and deep learning models right now. So it's a really natural way to provide a Python SDK. You know, we support it from day one and we will continue to support it for a long time. I'm curious with the kind of extending that just a little bit, beyond being in the language, I like the notion of the friendly language.

20:44The word friendly appeals to me in terms of that user experience. Could you talk a little bit more about kind of why you're branding around friendly and what that means from a user experience standpoint? What other aspects of it make it friendly? There are so many things out there that are not friendly that that definitely grabs my attention. Yeah, absolutely. I think what would be nice to explain is for the past 10 releases, we have dramatically improved the friendliness of our framework. Hopefully, I hope that's the experience that people will get out of this. The main point is to reduce the cognitive load of any developers who want to use our framework.

21:33So I'll give one concrete example. We introduced the Flower CLIs a couple of releases ago, I think probably late last year. And what this does is with a simple FLWR space, new command, N-E-W, a user is able to navigate these options through the command line and immediately have a templated project to work with for federated learning and it runs out of the box. After Flower Neo, the user goes through this, just follows the steps and then you do FLRR space run and it runs out of the box. And we have you know the core templates that is necessary for users to build on. We have the PyTorch, Trans-A-Flow typical ones and the more exotic ones you have Jacks and those who want it you They can use NumPy as well.

22:21All of these provides the boiler code for use to get started with, and it reduces so much startup time. Then with that, once a user has built all their applications, the user can also really monitor their runs. We also introduce commands like flr space, ls. It's really like ls in your terminal to just see what runs, what Webflow runs are running at the moment. and also others like FWR space log to see the logs of your code. So all of these really simple CLI tools really help a user navigate and work with running code much more easily. Previously, I would say, you know, 2021, 2022, early 2022, the flower framework was in a different place.

23:12How it worked back then was, it was still friendly, But the way that a user would need to start the federation would be to start three Python scripts. And this is not as intuitive or natural if you want to scale up or put into production. So with the introduction of the Flower CLI and a different way of deploying the architecture which drives the federation, it really makes it so much easier for users to start building and then deploy the code. Well, you were kind of leading into maybe what was going to be my next question. You mentioned kind of taking things into production. So some people might hear kind of friendly framework, which is a good thing, as Chris mentioned, but they might associate that with prototyping and learning and that sort of thing, not necessarily production usage.

24:12So I'd love if you could kind of help us understand what does a, if I'm implementing a federated learning system with Flower, what does a production federated learning system look like? I'm sure there's different sorts of ways that that could manifest, but certainly you've seen a lot of use cases. Maybe you could just highlight some examples for us. What does that production federated learning system look like? And what are some of the considerations that you have to think about going from kind of a toy prototype of like this might work to a full scale production rollout? Yeah, absolutely. I think it is a nice segue between the fairness aspect and moving to production.

25:01Because what I just want to mention here is that I walked through a very simplified workflow of how a user would build out an FL solution. With the flower framework, you could build and write the apps that you need for your aggregation, your server aggregation, and also for the clients, which actually train the models at the data sources. In the first iteration, a user might actually run it in what we call the simulation runtime. So without worrying about the actual data sources or to work out the data engineering aspect of it, you could test the implementation of the basic architecture in the simulation runtime using datasets that are obtained from hugging face, for example, or from datasets that you could just create artificially just for task purposes.

25:52With the same code that you use to train the models and the clients and the aggregate, you can then point the code to a different runtime and then execute it in what we call the deployment runtime. And this brings us one step closer to production. so once you have this mode of execution the clients would then be tapped in to the data sources and you can then start training your actual federated model so what does it take to deploy a production system all right so firstly there is a nice acronym that i like to use from the tiny ml community it's blurb i'm not sure if you've come across that before have you come across that before this curiosity yeah yeah but go ahead and uh yeah go ahead and explain yeah yeah yeah so the tiny ml community talks about the bandwidth latency efficiency reliability and privacy i'm not mistaken i could be wrong with the last one but in the production grid system what you really want is the reliability of the deployed solution to do to do the full computation It doesn't have to be fair to learning, but systems in general.

27:04So with the current version of the Flower Framework, we have separated what we call the application layer where users will build in apps and these are the ones that users will modify. And then we also have the infrastructure layer, which underpins this system. This infrastructure layer is responsible for receiving the flower commands from a user and then to distribute all the necessary code to the clients for the clients to actually perform the training. So in flower terms, you come across it, but we call this the superlink to actually host the server. And the supernodes are the long running services, which basically orchestrate the clients.

27:56So these two components are long running. So with these two components, because they're long running, the users can then run multiple, execute multiple federations across other systems without worrying about any of these components failing. So this is where the reliability comes to the picture. Because the connections are also established, we also handle the bandwidth and the connection so we try to reduce the latencies between the super notes and the superlink as well so the infrastructure is something that's being deployed once and that will persist for the lifetime of the project and this makes it much easier for users to continue to work with the production grid system so it's always there waiting for you anytime a user wants to go in and execute a run and look at the results is always there without worrying about any competent failing and stopping the run

29:09chris and i are so happy that you are joining us each week to hear from amazing guests and listen to some of our own thoughts about what's happening in the AI world. But things are just moving so quickly. And we recognize and want to see you all participate in the conversation around these topics. That's why I'm really happy to share that we're going to start posting some webinar events on our website, where you can join virtually and actually participate, ask questions, get involved in the discussion, and really deep dive into various topics related to either various verticals or technologies or tools or models.

29:53The first or next of these that's coming up is June 11th at 11 a.m. Eastern Time. It's going to happen virtually via Zoom, and you can find out information about that at practicalai.fm slash webinars. This is going to be a discussion around on-prem and air-gapped AI, in particular, as how that relates to manufacturing and advanced logistics. I've seen personally, as we work with customers in this area, just the transformative power of this technology there, from monitoring machine-generated events to providing supply chain advice and so much more. But there's a lot of struggle in terms of deployment of this technology in those air-gapped or in those on-prem environments.

30:44So that's what we're going to talk through on June 11th. I really hope to see you all there for this discussion. Again, June 11th, 11 a.m. Eastern Time. It's going to be virtual via Zoom. And you can find out more information at practicalai.fm slash webinars.

31:26so chong i love this idea of the sort of super notes and and super links and my thought is I'm trying to work out in my head kind of if I was, you know, let's say I'm working in the healthcare space and my sort of nodes are maybe different hospitals or different facilities in a network or something like that. And I have a central place where I have my super link and I'm doing the aggregation. Just from a practical standpoint, as I think Chris mentioned before, Or you have these different facilities, you have different maybe stakeholders with different data. What do I need to do as like, let's say I'm the person that's in charge of running the experiment, training the model.

32:13What do I need to do on the setup side to sort of connect in these super nodes or wherever the clients are? What sort of needs to exist there? How do I kind of register them and that setup process to really get going before I'm able to, you know, go in, like you say, and from a user perspective, run experiments or perform runs, training runs and that sort of thing? Absolutely. So there are many ways to go about it. But I think the cleanest way is to think about two groups of roles. One is the administrator role. and they are responsible for deploying the super nodes in each of these, let's say, healthcare facilities, healthcare centers.