Deep learning in Rust with Burn 🔥

0:07Welcome to Practical AI. If you work with artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners for helping us bring you practical AI each and every week. FASI.com, fly.io, and typesense.org.

0:42What's up, friends? There's so much going on in the data and machine learning space. It's just hard to keep up. Did you know that graph technology lets you connect the dots across your data and ground your LLM in actual knowledge? To learn about this new approach, don't miss Nodes on October 26th at this free online conference. Developers and data scientists from around the world will share how they use graph technology for everything from building intelligent apps and APIs to enhancing machine learning and improving data visualizations. There are 90 inspiring talks over 24 hours. So no matter where you're at in the world, you can attend live sessions to register for this free conference.

1:18Visit neo4j.com slash nodes. That's N-E-O, the number 4, J.com slash nodes.

1:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? I am doing very well today, Daniel. It is fall weather out, and I'm enjoying getting outside. It's fall. It's raining here today. Yeah, it's a little cloudy out, but I'm enjoying it. It's nice weather. And so, you know, it's like part of me wants to stay inside and do the fun things, like especially about what we're going to be talking about today. Right. And part of me wants to get outside and enjoy the weather.

2:18Well, it's that time of year where you just want to curl up next to a fireplace and burn some firewood. Oh, my gosh. You took us right there. I'll tell you what, before you say that, I'll just say this is an exciting episode coming up because I think this is a little moment where we're going to talk about our industry maturing a little bit through one effort. And with that said, I'll let you go ahead and do the intro. Well, the connection to Burn is because Burn is a deep learning framework that's built in Rust. And today we have with us the creator of Burn, Nathaniel Samar. Welcome, Nathaniel.

3:00Hi, thanks for having me. Yeah, well, I admitted to you before the episode that I am basically uninitiated in terms of Rust goes. I've looked at various articles. I think that I've run Rust programs just in a sort of hello world sort of way. Probably my biggest use of Rust has been using Rust in the Python linter called Ruff, which is really great. So that's kind of a circular thing. But for those others out there in our audience that might not be as familiar with Rust as a programming language, could you just tell us a little bit about this sort of like, what is Rust and why Rust? Yeah, Rust is, I think it's falsely being categorized as a low-level programming language, probably because of the historical reason.

3:58But it's very general programming language that can be used for high-level stuff as well as low-level stuff. So the main reason to use Rust is maybe when you need to go through multiple abstraction boundaries without having to pay for performance. So yeah, this is how I defined it. And I could be wrong about this, but I think one of the great features, along with Go having a really great mascot, we've got, isn't it a crab? If you see crabs or something for Rust, isn't that a thing? Yeah, I think it's a cute crab. It's a cute crab. That's the mascot, yeah. I think it's important for programming language to have that.

4:41Yes. You have Python for the snake with this programming language. We have got, I don't know what it is for Go. It's the Gopher, the Go Gopher. Yeah, it's quite nice. Yeah, the Gopher. It's funny, and you mentioned that. Go is actually how Daniel and I got to know each other. We met in the Go programming language community, and we were kind of the two data-oriented people at the time. This is going way back. There are many, many data-oriented people these days, but got to know each other. Subsequent to that, I had been hearing about Rust for a while, and I got very interested in it, not only because, as you pointed out, it's a fantastic general-purpose programming language all around, but it also does have a lot of really amazing low-level features and performance capability that attracted me to it.

5:29So I'm not nearly as accomplished in the language as you are, Nathaniel. I still love Go, but Rust is now another programming language that I have fallen in love with. Yeah, I think Go is really well suited for web services. So we've got a lot of tooling around that. It's really pragmatic to use it for that stuff. So yeah, Rust is getting there, but we've got the whole async stories behind that. Yep. And for Rust itself, you mentioned kind of people have this stereotype of Rust as a low-level programming language, but could you give maybe some examples of the types of things either you've built in Rust over time or that are possibilities just to kind of give people a sense of what people are doing with the language.

6:13Obviously, we're going to be talking about deep learning, which is, thanks to you, something that can be done with the language. But what are some of the other things that are out there that people are doing right now with Rust? Well, I think it was first created as a replacement for C++ to write browser engines. So this is maybe why it was known as a low-level programming language. but now I think it's used in game engines. It's also used to do web frontend so you've got like Lettos and Deoxys which are frontend libraries like React and Vue so this is pretty high level. We've got also command line libraries that you can use like metaprogramming so that it's very easy to do your command line arguments all of that kind of stuff.

6:58So yeah, there's tons of things that are built with Rust high level and low level So you can mix and match your own applications. Of course, there is like the web services with Tokyo, the Async Front Time, Action, a lot of if you want to do web services. There is also libraries for that. Yeah, this is a project on top of my mind. It was one of the first languages that really embraced WebAssembly and got it out there. it's interesting uh coming speaking as kind of a novice in the language and coming from most recently go there's always this debate on go versus rust that you tend to see in articles out there and i i really found room for both of them um and i go back and forth at this point i will point out uh you know whereas go is one of those languages that has runtimes that kind of manages memory rust has a really cool feature to it it's not specific to what we're talking about today but the compiler ensures that you don't have memory faults seg faults which is something like 70 % of all the bugs in software according to Microsoft and so it has a really interesting way of approaching ensuring that you can produce bug-free software or at least much fewer bugs in it far fewer bugs in it so it's a pretty cool language I'm just curious as we're talking about the language in general what's your favorite feature what are the some of the things that made you turn to rust versus some of the other languages you may have worked in?

8:21This is hard to just choose one feature. I think this is the whole package. Said like a real Rust aficionado there. Yeah, but like my favorite feature is not the reason why I started writing in Rust. But now I think my favorite feature is just associated types because it can abstract data types, like something that is really hard to do with other languages. So, yeah. And could you explain a little bit of like when might that be useful or how is that useful in terms of like when that might come up in your programming? Well, it's when you need to abstract the type you're going to use, but you let the implementation decide the types.

9:02Normally, like you have the generics, the generics you have to, maybe you have a list and you have to say, okay, I want a list of string. But it's when you use the list that you decide the type where associative type is, okay, I've got maybe a list, but I don't know of what. It's the implementation that decides of what's going to be the list. So sometimes it makes sense. For instance, in Burn, we've got a backend, a backend trace, which we can have multiple implementations like CPU, GPU. And we have associative types for the memory, for the tensile endpoints, for all of those things that you can manipulate at a high level, but you don't have to know which type it is.

9:41It's to the implementation to decide. I'm just going to ask maybe like an ignorant question but I think maybe some people out there might be wondering it. If I'm working in Python, this is a language where I don't have to compile my Python code. Some of the things that we're talking about here with the compiler and other things, a lot of people don't think about, although there's some intersection with that. So could you describe when you're writing a Rust program. What does that look like in terms of is it a statically typed language? You're talking a little bit about type there. It sounds like you talked about a compiler.

10:22So am I right in that it's a compiled program and then you can run the binary on some architecture? What is it like to work in Rust as compared to something that people might be very familiar with like Python, where a lot of people that are probably listening to our episode or have their Google CoLab notebook pulled up right next to them, right? And they're doing all sorts of things with a Python interpreter. What is the workflow and programming like in Rust as far as how the language is set up and how you work with it? Obviously, it's a bit different than working in a notebook. Like you said, it's the strongly typed static programming language similar to like C++, Java, all of those older language.

11:08So for people that comes from Python, maybe you're aware of the Python type int that you can use. It's a bit like that, but you have to use it everywhere in all of your functions and definitions. And the workflow, something that I like is that in Rust, I think it's one of the only programming language that does that, is that when you write a function, you can just write the test below it. So that's kind of the way where you can get some feedback on what you're actually writing. And it encourages good practice because you're writing a test that can be reused all the time. It's not a script that you're trying to just run on the side.

11:45You can actually commit that and it describes how the code should run. And that's how you get interactivity with this. And since you have a packet manager, which is Cargo, it's pretty easy to just execute the code you're writing. To follow up on that, Cargo, the package manager, is based on a lot of the best practices we see in some of the other programming languages. For instance, in JavaScript and the Node community, you have NPM and there are several others. And the Rust community really drew from kind of best practices on that. Another thing to kind of follow up on the compiler notes that Nathaniel was mentioning was a lot of Rust developers kind of see the compiler almost as a pair programming partner in a sense to where instead of just hitting compile from time to time like you would in Java or something like that, the compiler is so comprehensive that it kind of helps you and you kind of use it to write the right code and you get to the end of the process and know that your code will actually work without runtime errors.

12:46so it's a different way of thinking about being a developer it takes a little bit of a mind shift to adjust over to it this is very different like in python an important skill is just to be able to read the stack trace because you're going to have a lot of exceptions when you run your programs and you you have to learn how to develop your program this is kind of a hard skill you have to do when you learn python in russ it's you have to learn how to write the compiler errors but they made at least they tried to make it as easy as possible even sometimes you've got links to the documentation it opens a browser it can read why you have that error it explains the reasons why so this is a different set of skills and yeah this is quite different from the workflow you use with python maybe just one more question about rust in general before we dive into some other things.

13:40What is the Rust community like in terms of whether it be, is there active channels where the Rust community communicates with one another, conferences, meetups? What is the Rust community like? And is it growing? How is it changing over time as you've been with the language for some time? How has it developed in the time that you've been part of the community? I'm not sure about all of the community, obviously, but I think it's pretty friendly. Like there are some Discord channels where you can just go and ask your questions if you want to. There is an active GitHub issue. So the language is open source.

14:20If you have a problem, just open an issue and people maybe are going to help. So this is a pretty inviting community, I think. This is part of the reason why it succeeds, I think, because if you don't answer questions, you don't help people used, your technology doesn't really work out. I never went to a conference for Rust yet, but I know there are many, so maybe I'm going to go to some later. You know, one of the topics that has been kind of a recurring topic between Daniel and me over a number of episodes, we've been tracking kind of the maturity process of the AI community and kind of what it takes to kind of level up and to take it to the next level.

15:04And on a number of different occasions, we've talked about the fact that if you look at other communities that have arisen before this one, often it takes kind of broad support. Whereas in the kind of the early days, you know, that we're really still in, in my view of modern AI, it has been largely dominated by a single programming language, which most of our listeners are very aware of, which is Python. which has really been kind of the focus of where all the work is. It's where all the APIs have been focused on and everything. And we've discussed quite a bit about how for AI to mature, it needs to become more broadly available to other languages.

15:44And so that as you have different types of use cases addressing, you know, different business needs, and that requires languages other than just Python all the time, how do you get to AI and what kind of bridging do you need to do? it leads me nathaniel as i wanted to ask you is it's clearly a need that the community has had to be able to start getting rust and other languages in there i'm curious how did you approach this what was it about trying to get rust working as a framework that could work with ai tools of the day how did you get into that what was your motivation what did you see as the need at a personal level?

16:23Well, I started working on Bird because I was experimenting with asynchronous neural network and I wanted to make something a bit not standard, let's say that. And I needed like multi-trading, concurrency and stuff like that. And it was really hard to do with Python. And I have a software engineering background. So I said to myself, well, if it's hard for me to do that, then maybe it's too hard for any researcher to do that. So that's why maybe we don't have yet and architecture for that kind of stuff. So I say, well, let me try and make a framework in a language that has support for high-level programming and concurrency and all those things.

17:02And yeah, it's pretty much the description of Rust. So that's why I started writing a framework in this language. And then it just was a personal project for a long time. I just was experimenting with it. And yeah, it grew with time. When you first started thinking about Burn and these problems that you were looking at, what was the current support for doing, whether it be kind of quote unquote traditional machine learning, like random forest, SVM, whatever that is, all the way up to kind of deep learning in Rust, what was kind of the state of things? I'm looking at your burn repo at IC. You've at least been submitting pull requests since July of 2022.

17:48too. Maybe I'm sure some of it goes back further than that. So back to those days, what did the ecosystem look like in terms of its support for these things? Well, I don't think there was a lot of deep learning framework in Rust. So there were some experiments, but nothing really pragmatic that you can use. So I think there was a library for normal like SVN random forest in Rust. I never used it. But yeah, I don't think it's comfortable yet to Sci-Chat Learn and PyTorch, which is very complete. It's interesting because some of the sort of early stuff that we were doing in Go was similar there.

18:29There were certain packages like for whether it be kinds of regression or hypothesis testing, statistical things, but not really a robust deep learning framework. One of my questions would be in Go, I know one of the struggles with trying to support really robust deep learning is not necessarily the fact that you can't create a nice package with a good API. but a lot of these sort of specialized libraries and toolkits like CUDA and GPU support make things a little bit more difficult so it might not be that but what what did you see at the time you started working on burn as the big challenges on the rust side and has that been the case as you develop the package or have other things become the kind of dominant challenges over time?

19:33Yeah, all of those things are hard to work with, like CUDA, having your own GPU kernels, all the drivers, not necessarily easy to install on other platforms. There is a GPU libraries in Rust that has write kernels. This is like WGPU, so it's targeting the web. But when I started working on burn i acknowledged that it was pretty important to be generic over the back end so that we can write the best back end for the specific hardware you're actually targeting because it's probably always going to be faster to write cuda for nvidia to write like low level c or rust maybe with SCMD support for CPU or to write with the metal graphics driver for Mac.

20:21So I was aware that one backend cannot be written for all of them. And I just defined the API and I just used LibTorch as a backend because there was already bindings to LibTorch and Rust. So this allowed myself to iterate over the abstraction over the user space API and not necessarily worry about speed and writing all of the kernels, just getting the abstractions in place and the software architecture in place. And it's more pragmatics. It's probably like as fast as LibTorch by default. And then I can just go and write more kernels afterwards, which is what we're doing right now. I'm curious, do you feel, given the low-level capabilities capabilities that Rust brings to bear that so many other languages don't have?

21:12And that when you're looking at whether it be GPUs over time, and I know you're talking about, you know, using LibTorch in this case, but do you think that as you move forward, that that low level capability that you have in this language that other languages don't bring to bear will be a helpful, you know, part of kind of developing it and maturing burn over the years ahead? Does that low level give you an advantage that you might not have with other languages that we're trying to integrate in? I think so. Mostly in the part where we need to handle memory. So that's an important part of deep learning frameworks.

21:47You don't have to waste memory. We can leverage all of the type system of Rust to actually do graph optimizations and all of that kind of stuff that we're going to work on soon. And I think it's going to be easier to work with Rust to do that with good performance then it will be with maybe another programming language with the garbage collection because it has fine control over the memory. So not necessarily to write GPU kernels. When you do that, you're actually writing compute shaders. So it's not relevant to Python or C++ or even Rust. But if you want to handle memory and write the optimization pipeline, then I think Rust can be really useful.

22:27And just to get a sense of kind of the current state of burn, what is possible in terms of support and what you can do right now and what are some of the highest requested things that you would like to work on but kind of aren't there yet I don't know there are so many things that I want to work on time is just is limited so quite hard what I'm I'm really excited to work on is kernel fusion and really optimize the compute pipeline with lazy evaluation. So that's something I'm really excited to work on. Could you dive into that a little bit and kind of what that might mean for a user specifically?

23:14Yeah, in terms of user, it's just going to be faster. So this is really like optimization techniques that the framework, the deep learning framework can use. So yeah, there isn't a lot of impacts in terms of user API and usability, but it's just going to be faster. Gotcha. And would you say that right now, in terms of what people are doing with the package, now you mentioned that part of what got you into it was building kind of experimental models or architectures that maybe you were experimenting with on the research side. so I'm wondering with this package what are you seeing as the people that are using it what are they most doing with the package is it that sort of experimental research implementation side is it taking models that aren't maybe experimental and embedding them and you know Rust applications where they wouldn't have been able to before is it something else what are you seeing in terms of what people are doing over and over again?

24:20I think a lot of people are using it because it's easy to deploy on any platform because we have different backends so you can deploy on WebAssembly, you can deploy on even a device without operating systems. So this is pretty great in terms of deployment flexibility. But even though I started the framework because I had like a research idea I wanted to do, the goal of Burn isn't necessarily to be only for research. I wanted to go with kind of a blank sheet and thinking about all the constraints and who is going to use the framework. So I'm always thinking about the machine learning engineer perspective, the researcher's perspective, and then the backend engineer's perspective.

25:01So the one that is going to write the actual low-level kernel code and CUDA kernels and stuff. So there are kind of different user profiles or use cases that you can assign to the framework. Yeah. Kind of as a follow-up to that, is you were looking, and I noticed that you had quite a few people that were making contributions. For being a relatively young project overall, you have a lot of people involved in it. So, I mean, it looks like it's really getting a lot of traction. How do you kind of organize the work around it and kind of satisfy the interests of each of those personas along the way?

25:38Is there one that tends to lead or do you tend to try to have certain people that do different ones? How How do you approach that? To be honest, I'm not sure. I think the key is just to be reactive. So if there is an issue, just go and comment it. If there is a bug, try and go fix it. And I think the most important work I can do is in terms of architecture, like setting the stones in place. But then if I want to extend, maybe add more tensor operations, or if I want to add more neural network modules, then I can open issues. And people that are interested and can just assign themselves and actually do a pull request.

26:15And I just have to be really, really conscious about that. Do code review correctly, be kind. And I think that's pretty much it. I don't have any other secret. So Nathaniel, I've deployed a lot of models as part of my day job. Let's say that I am interested in Rust and I am interested to maybe take some model that I might have experimented a little bit with in a collab notebook or something like that and I want to make it, like you said, have the support for multiple backends implemented in a maybe more efficient application. What would be the process that someone would have to do to, let's say, get one of the kind of popular quote unquote models these days up and running in Rust using Burn?

27:11Is that something that's possible right now? How are people kind of pushing the edges with respect to that? Well, I think there are two different strategies. So we're actually working on being able to import Onyx model. So if you have maybe an image classification models, then maybe our end port is going to work. It's still in WIP, but if there is no crash, it's going to work. Not all operations are supported. But maybe for other models, you maybe need to write the model from scratch using our framework and then translate the weights and you would be fine to deploy it. So it's a bit of work, but working it with burn is quite intuitive.

27:51So the API is similar to PyTorch. The modeling API at least So it's not that hard depending on obviously the size of the model and the complexity of the model. Yeah, and I think I saw a few on the repo that people have already sort of done this. What are some examples of some of these that people have brought over into Burn? Yeah, I think there are community models for LAMA, for Stable Diffusion, for Whisper. This is thanks to the community. I didn't actually quote those models. but yeah since it's open source I think if you actually do the work to port maybe a model I think it's great to share it with the community people can start using it so yeah we have a few but we would like more yeah so call out to the listeners out there that are Rust people in the audience check it out and submit some of your own model implementations that that's a great way to contribute, I'm sure.

28:49You mentioned it having a similar API to PyTorch. And I'm kind of looking through some of the documentation here. I'm wondering if you could just comment on a few of the things that you call out as far as features of burn and kind of explain what you mean by some of those things. I think we already talked a little bit about the customizable, intuitive, user-friendly neural network module. So this kind of familiarity with maybe a PyTorch API, Maybe there's more to that. But you also mentioned this comprehensive training tools, including metrics, logging, checkpointing. Could you describe that a little bit in terms of what the thought process is in the framework around these things, which are definitely important practically, as you said, for the machine learning engineer, for the actual practical person who's trying to build models?

29:42Yeah. And even the researchers, sometimes they don't want to actually write all of the training loop. if that's not the core of their research. Yeah, there is a library which is called Burn Train, which try to bring a training loop to the user so they don't have to write it. So you've got like a basic CLI dashboard where you can follow all your metrics. You have your logging. So if you want to maybe synchronize the drive to maybe a Google account, you can probably do that. So it's similar maybe to PyTorch Lightning. so for the Pythorch users that are familiar with the projects but we also have that for Bering and we just have that.