#183 Will Falcon: Lightning Studio, an iOS for AI Developers?

0:28What does that iPhone experience look like? let you code on the cloud and stays on the cloud. You can still connect your local IDE and code, so it still has the local experience, but everything's on the cloud. Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I speak with William Falcon, the creator of PyTorch Lightning and founder of Lightning AI. Will introduces us to Lightning Studio, an innovative cloud-based platform that integrates various AI tools and frameworks, allowing users to train, fine-tune, and deploy models at scale. Will talks about his passion for open-source AI and democratizing access to powerful AI tools.

1:12I hope you find the conversation as useful as I'd. AI might be the most important new computer technology ever. It's storming every industry, and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power. So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, and of course, nobody does data better than Oracle.

2:03So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber 8x8 and Databricks Mosaic, Take a free test drive of OCI at oracle.com slash I on AI. That's E-Y-E-O-N-A-I all run together at home and work. We all know one person who's password challenged. Sticky note reminders, emailing passwords, reusing passwords, using the word password as their password. Because data breaches affect everyone, you need one password. 1Password combines industry-leading security with award-winning design to bring private, secure, and user-friendly password management to everyone.

2:56Companies lose hours every day just from employees forgetting and resetting passwords. A single data breach costs millions of dollars. 1Password secures every sign-in to save you time and money. 1Password lets you switch between iPhone, Android, Mac, and PC with convenient features like autofill for quick signups. All you have to remember is the one strong account password that protects everything else, your logins, your credit cards, secure notes, or the office Wi-Fi password. One password generates as many strong unique passwords as you need and securely stores them in an encrypted vault that only you have access to.

3:42I use 1Password, and you should too. 1Password's award-winning password manager is trusted by millions of users and over 100 ,000 businesses from IBM to Slack. It beat out 40 other options to become Wirecutter's top pick for password managers. Plus, regular third-party audits and the industry's largest bug bounty keep 1Password at the forefront of security. Right now, my listeners get a two-week free trial at 1Password.com slash IonAI. That's IonAI all run together, E-Y-E-O-N-A-I. That's two weeks free at 1Password.com slash IonAI. 1Password.com slash IonAI. Will, can you introduce yourself? Tell us about how you got to Lightning AI, and then we'll talk about Lightning Studio.

4:43Yeah, so I'm William Falcon. I'm the creator of PyTorch Lightning and founder of Lightning AI. And yeah, I came to AI from research, right, like most of us do. But before that, I actually was in the U.S. military, so I spent about six years going through Navy SEAL training in San Diego. and um i during that i got injured and so you know the navy gives you a few options at that point right they say the option to continue and finish was not one of the options they gave me um as an officer you only get one chance at training so that was unfortunate um but then you know they gave me the option to become an intel officer or leave and try something else so i left and you know i was considering things like the cia and like other other special operations and then just kind of found my way.

5:27It was like around the time when iPhones were coming out and iPhone apps. And so I started building, just started coding, coding apps and eventually found my way into computer science and math at Columbia. So don't ask me how, but I ended up doing my undergrad at Columbia University in New York. And then there I learned that you could like major in computer science, right? And so I did. And then - Did you know a guy named Chris Wiggins? Yeah. So I know Chris from the Center for Data Science, I think is where I know him from there. And yeah. And so right around the time I got to Columbia, it's like 2013, which is when deep learning had just taken off like the year before that.

6:06I had no idea what it was. Right. But like I show up to one of my classes and there's this professor, Tony Jabara, right, who like ran Netflix and stuff for a while on the deep learning side. And he's teaching neural networks. And I'm like, I don't know what any of this is. And in one of the demos he gives is MNIST from Jan down at NYU. you right down the street and uh and at the time i was like it like it had like a little carousel music and i was like okay i don't really know how that's useful right but but it was interesting and so i started learning about it and long story short about two three years later i ended up doing research in computational year science and um and in 2016 17 i want to say i went in europe's and um in montreal and um that's when you know back then it was like a few hundred people so i discovered the deep learning community and somehow ended up kind of going into that.

6:54I had taken up a job at Coleman during that time. So I told my manager, hey, I want to do deep learning the training floor. And they were like, no. And I was like, fine, I'll leave. That's all right. I went back to school and just did research. And then at that time, I started building internal software for myself to be able to move faster through it. And that's why I eventually opened source, which is known as PyTorch Lighting today, right? So it started super, super early. And I ended up doing my PhD down at NYU with Yalakun and then Kyunghyun Cho as well. And then about three, four years into it, I dropped out to Starlighting AI.

7:28PyTorch Lighting took off. I was a Facebook research at the time. And then, you know, thousands of companies were pinging me. They're using it. They're trying to put models in production. This is like late 2019, early 2020. And, you know, they were super encouraging and they basically were like hey like you should probably pursue this and see how it goes and uh you know we'll support you and they have been very supportive and so i left uh facebook and i have been focused on lightning since then and um you know at the time we were trying to figure out how to remove a lot of this like overhead that you need to know to like do deep learning right like back then you had to implement your own gradients and all these different things so pytorch lightning solved pytorch for meta solved a lot of these problems and then lightning solved the scaling problem.

8:09But then you still have to know Kubernetes and Docker and like cloud and a million other technologies. And so I started trying to figure out like, how do you do that? Like, how do you, it really felt like MS-DOS or like a Blackberry before an iPhone. You're like super clunky. And so I was like, well, what does that iPhone experience look like? Like, what does that look like? And so it took us about three years to figure it out. We tried many things. And ultimately we landed on studios, which we launched. It's been an industry behind the scenes for about a year, like being used by enterprises, but we made it public a few months ago.

8:42And I actually do think that studios achieves that iPhone leap where it is super usable. If you go to Twitter today and LinkedIn and social media, you hear people talking about it like revolutionary. It's the next thing. That's what we wanted. And that's what Lightning does. We wanted you to get that 10x experience on something. Let's go back. explain what PyTorch Lightning does sort of in layman's terms. Yeah. So PyTorch is a library built by Meta, right? Or Facebook at the time. And what they solve is the ability to do computational graphs, right? Like when a model learns, you have to compute gradients and you have to update those gradients.

9:21So it's like how to express a math function in the program in essence, right? So they give you the raw tools to do that. So it'd be me like giving you a bunch of tools to build a car. Like here's a wheel, here's this, here's an engine. But then you still have to build your own car. And so people come up with ways of building their own cars and they'll kind of look and feel the same, but they're not standard. Some people put like four wheels, five wheels, three wheels. They put them in different places. Byter's Lightning came in as an interface, like a front end for that that says, no, here's how you do it.

9:46Here are the standards. You put wheels here, you put this here, you do this here. And it structures your code. It's not an abstraction. It's really just structuring the code and organizing it for you. And then from that, you're able to scale. And that came out of many years of research. And the big deal that this unblocked was the ability to train across thousands of machines. And in 2019, that wasn't happening. At Facebook, it was me and two other teams who were doing this. I was an intern at the time. And the other teams were professional engineers, like 10 of them, doing these things. And I got a lot of their help into building Lightning into what it is today, putting all their knowledge into the programs.

10:23And so that's how I came out. And so in 2020, when I left Facebook, I think the paper we published a year after, we trained that on like one or two thousand GPUs. Right. And that was in 2020. Today, people are like starting to learn how to do that. But like we've been doing that for a long time. So it gives you that scaling ability. Right. So if we're used to like web frameworks, we basically say like PyTorch is JavaScript and PyTorch lighting is React. Like you wouldn't build your own React library. Once that product had traction, you started looking around and seeing all the other tools and frameworks that you need to work with PyTorch or with PyTorch Lightning.

11:05And you eventually brought it all under one umbrella in that studio? Exactly. And so just like in web, right? I mean, actually, I argue that even AI is a lot harder because you have so many moving parts. But PyTorch Lightning just does one piece, but you needed so many things. So Lightning was the first thing to integrate other tools. So we were first ones to integrate the ability to train on different hardware. So before Lightning, you didn't really code your stuff and then go to GPUs and then go to TPUs. Today you do that, right? That's something that we pioneered in Lightning. And so we started integrating different accelerators, which is super valuable today because there's not enough GPUs to go around.

11:44So you kind of have to be able to try other hardware. Then we also brought in things like experiment managers, like TensorBoard, Neptune, Weights and Biases, Comet, all that stuff, right? Integrated into this. So it started growing into kind of like an operating system in essence, right? That connected a lot of things together. And that was kind of the premise behind a lot of what we did. And in Studio, we basically took that farther. We just said, okay, now we're going to bring in your cloud infrastructure as well. And then all your teams, your developers, your data scientists, your machine learning engineers, your designers, Everyone can work on a single thing together from the browser, but the infrastructure is at scale, right?

12:21And last time we spoke, I had just spoken to Vercel, which has services or a platform where you can build a model and then access different foundation models. It's kind of like Bedrock at Amazon, maybe more versatile and with more options. And I saw Vercel is in Lightning Studio as one of the tools or platforms that you can access. And it's a little confusing to me. Like, what is the, you know, you keep on getting these umbrellas that then give you access to things further down or further upstream, I guess. Can you talk about this as sort of hierarchy of platforms or models or umbrellas? Yeah, so I'm not super familiar with how Vercel works, but from what I know, it's mostly about web development and deploying web apps, right?

13:29Like that kind of thing. That's not really the focus of Lightning, right? So what you saw in Lightning is the ability to deploy web apps. You can deploy React, you can deploy Angular, Next, et cetera, but that's not really the focus, right? The focus of Lightning is how do you deploy models at scale? How do you train models? How do you fine-tune them? How do you take 10 terabytes of data and do distributed processing on it? So it's really more about the scale. What separates Lightning is this ability to take your studio and scale it to a thousand machines instantly. That's not something that a website ever needs.

14:01A website is happy to work on a single machine and maybe scale a little bit, but not at the scale that you need for deep learning. right so so it's a fundamentally different thing um yeah i think all tools have their place for sure right and we always happy to integrate with things um but definitely the main the main thing that we're going after is the scalable workloads uh the machine learning uh kind of side of it and like through that people do have to make web apps and they have to make websites so we give them the ability to do that pretty easily as well but you know our customers makes tools all the time they don't just use one yeah when you say to to build models you're talking about any machine learning model?

14:38Are you talking about foundation models or generative models? So PyTorch Lighting came out before foundation models were a thing, before Gen.AI was a thing, before any of this, right? So PyTorch Lighting, in fact, one of the foundation models, Stability.AI, is built using PyTorch Lighting. So from the very beginning, we've been at the epicenter of Gen.AI. So to answer your question, our tools scale from the simplest models, like literally tiny, tiny models to foundation models on thousands of gpus you know over the years to our frameworks and our tools have to the stood the tests of time and we know that they actually do work for pretty much any type of model today right and and that wasn't we didn't actually know that in 2021 when lightning came out first right so so that's that's been true but um yeah it's really any type of model um and um and i think we definitely do like gen ai and foundation models better and when i say build that's a good that's a good point like i don't literally mean you have to build the model most people just grab a model from open source right i don't mean that for for researchers like in fair or you're doing your phd like sometimes i do mean that and we do that but really the majority of people are fine-tuning models they grab something they drop it they put some data on it they fine tune it for a few hours that's great that's that's exactly one of the bread and butters that we do in the platform does does studio then allow you to do things like connective vector database or you know, and implement RAG and all of those things that people are doing today?

16:06Yeah. So we have like a public collection of studios that we put out, kind of like a gallery or like a library of these. These are templates, right? In fact, like just yesterday, we put out a RAG one. So that one, you can grab a whole data set, do RAG with it. And there's like a UI, you can query it as well. You can integrate like an open source database or your personal databases as well. And these things come up a lot. Pretty much anytime there's a use case, someone's like, oh, can you do this or that? Usually either we do it or someone in the community will put it together in a few days, but the platform is very extensible, right?

16:39It's kind of like having a car and asking if it can drive to Texas or the supermarket. Like, yes. This is open source, is that right? So a lot of our frameworks are open source. The platform itself is not, but a lot of the pieces of the platform are open source. but the code that we give you, the templates are open source. So if you ever, like that RAG template, the code is free in there. You can use it. You can take it off lighting if you want, right? But it already works out of the box, I think. Are there other things in the market that have approached this and you've gone one layer higher or broader that people maybe know and can relate to?

17:19So, you know, kind of like, remember when the iPhone came out, most things look like blackberries and palm pilots most things that were out there at the time razor phones right like that kind of thing that that's what that market looks like today and and that's because people were kind of trying to do the same thing just slightly better different ways right we said no we're not going to do that in fact we like stop we like i don't really care what other people are working on in general like we don't follow what what the market's doing we like we just talk to our users and hear what they want to do and we solve it our own way kind of like Apple, they'll solve it their way, however they want to do it.

17:54And no, so we really refought the problem from the ground up. And so we said, it's got to be fundamentally different. And pretty much every single tool in the world today forces you to code on your laptop, on your local machine, and then you submit to the cloud. And that's a paradigm that they've all adopted. We got rid of that paradigm. We don't let you do that. We just let you code on the cloud and stays on the cloud. You can still connect your local IDE and code. So it still has the local experience, but everything's on the cloud. And that's a very, very different paradigm. No one else is doing that.

18:26And when we started doing that, people thought it was crazy, right? Kind of like when Apple took keyboards off phones, they thought it was crazy, right? Yeah, yeah. I would imagine, though, that there are latency issues if you're working in the cloud. I just, my, you know, this is way before machine learning, but I used to have to work on, you know, to do the expenses of the New York Times. You were working on a remote platform. So everything you input, there'd be like a little incredibly irritating delay and then that sort of thing. So how do you overcome the latency issue? issue. We spent a lot of time making that go away.

19:14Right. And that's why I also said you can also code from your laptop. So there is no, there's zero latency when you connect your local VS code or IDE, it's exactly the same experience. You know, the market has gone through this many times and there it's always, and I've been in the other side too, where I've been skeptical about things. Right. But like you ever use, have you ever tried Figma, for example, for designing? Figma is like a browser design tool, right? It competes with like Photoshop and those kinds of tools right now when figma came out you know you ask kind of designer and they're going to be like wait like design on the browser why like it's not powerful enough i have my photoshop on my laptop right adobe bought figma for 20 billion dollars why right because it is that is the future most things are moving that way you just have to be very good about how you do it and our team has spent a lot of time trying to solve these issues and i think we've landed there now it's going to get better 100 right the first iphone and the current iphone are very different but we have to take the industry and leap forward it that way and we have to say it can be done and we're going all in on it because we believe it is the future um and most people are i would say about three years behind now because of that right but we have a lot of knowledge on how to keep improving things as well yeah and then uh you integrate uh all of the like copilot or i mean github copilot or codex or all these other tools that people use while they're coding.

20:41So you're building by coding or you're building by putting together different elements that are pre-written. Yeah, exactly. So, you know, like an iPhone has iPhone apps, right? And you need apps to do your work, right? You need a way to write. You need a way to send emails, calculator, flashlight, et cetera. When you're developing AI, you need apps, right? So to us, everything is an app. So you want to train a model? There's an app for that. You want to fine tune a model? There's an app for that. You want to deploy a model? There's an app for that. You want to do RAG? There's an app for that, right?

21:21So every one of these things has an app embedded in it. Depending on what you're doing, it may be a full code experience. So all you want to do is code and develop. there's literally an id to do that uh but if you don't want to do that there might be like another ui like some of these um embedding things there's no code you're just like watching plots and dots on the screen right it's kind of like no code so you know just like your mac sometimes you write into an app and sometimes you don't you click into it right so once you have this built the studio built and it's got pytorch lightning uh and then all of these other apps uh are you are you continue that way i guess the the unique thing is is that this is uh all being done in the cloud or at least if you're doing it locally it's it's uploading to the cloud um what are the other they mean you then you have all these apps that you can use but what are the other features of lightning studio uh that are not covered by an app or is it really an orchestration layer to like manage all these different tools so so there are like features that are not apps that specifically right so like the first thing is like you can create one studio for every task so you know let's say you want to train a model you'll create a studio just to do that and then you'll create another studio maybe to do Rack.

22:49And then you'll create another studio maybe to do, I don't know, fine tuning and another studio to serve, right? You can have infinite of these. In essence, what you're doing is you're like taking your laptop. Imagine if like anytime you want to start a new project, you could just have a brand new laptop. That's what you have, right? And you can store that laptop. And then when you want to work on that, you just open that laptop and it's ready to go. So that's kind of the first thing I think is the most useful. Like I program in probably three or four things, you know, front end, back end, machine learning, data science, different times.

23:20I'm like, when I want a context switch, I just go turn on that studio and I'm there. I don't have to like set anything up, right? So it's a lot faster. That's, I think, probably one of the biggest features for everyone. It's just like, things just work out of the box. Then I can share that with you. So let's say you today, you're like, hey, how did you fine tune that model? Can you share that with me? I'll say, sure, duplicate my studio. And you literally take a carbon copy of my laptop and it's done. So, and then the community creates these and they share these. So you asked me about the rag.

23:47We didn't create the RAG one, someone else did. And now you can go take their RAG app and deploy yourself in five seconds. And you have to do zero work. They're RAG Studio. So that's one. You have the ability to work together. So let's say you and I are coding and maybe you're learning to code. You send me a link, just like in Google Docs, and I go and type with you and we just code together. So I can teach people how to code. I can help junior engineers onboard easier. There are a lot of things we can do there. You have the ability to measure costs in real time. So like you can say, I don't want to spend more than a hundred dollars on this thing.

24:20And you put a budget of a hundred and we will measure the cloud costs to a second. Ask any person in enterprise today, any company, how much they spend on anything machine learning. They can't tell you. You ask us, I can say yes. On that thing, we spent$50 ,000,$271. That's it. No question asked, right? It's all there. And then there's a bunch of team management stuff, like who can see what, when, at what levels of the org, who's an admin, who isn't, what data can they see? What can they not see? right the ability to connect to different data sources like snowflake and databricks data and then bringing data from s3 upload your own data right so all of that gets ingested so yeah lightning in essence is kind of like an operating system right it just becomes this center that everything kind of comes together and then you can create studios that are specialized to specific tasks that you need and you were saying that without lightning studio people would have to set all this up themselves.

25:15Can you walk me through a use case to build something and the steps that you would take and then how that is different with Lightning Studio? Yeah. So let's say Mixed Trial, I think it's one of the best models out there, came out. And this is the mixture of experts. And let's say you want to deploy that model, right? And for you to do that, you have an option. You can either go to one of these API companies and then hit a button and then they'll deploy it for you. But you are kind of subject to them. What if they go down? Like OpenAI goes on all the time, right? So what happens? Is your app going to stop working?

25:55Like your service is done, right? So do you really want to give that up? I don't, I'm not sure, right? So that's your first option. Second option is you go find the code in open source, and then you like download it to your laptop, and then you like set it up and try to get it to work. And then you like try to find a cloud machine somewhere. And then you like go to that cloud machine and you got to set it all up again and then maybe it doesn't work and you spend like a week or two doing this it's extremely hard right on lightning you go to studios there's the gallery there with all the templates you find the moe template you click on it you press duplicate takes about 35 seconds it's up and running and the model's already deployed and everything there and you have the code fully so you can you can delete the code if you want but it's fully transparent to you so you have full control over it you can hack it or you can leave it alone and Leave it how it is, right?

26:40AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power. So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application, development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, and of course, nobody does data better than Oracle.

27:28So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, take a free test drive of OCI at oracle.com slash IonAI. That's E-Y-E-O-N-A-I, all run together. At home, at work, we all know one person who's password challenged. Sticky note reminders, emailing passwords, reusing passwords, using the word password as their password. Because data breaches affect everyone, you need 1Password. 1Password combines industry-leading security with award-winning design to bring private, secure, and user-friendly password management to everyone.

28:22Companies lose hours every day just from employees forgetting and resetting passwords. A single data breach costs millions of dollars. 1Password secures every sign-in to save you time and money. 1Password lets you switch between iPhone, Android, Mac, and PC with convenient features like Autofill for quick sign-ups. All you have to remember is the one strong account password that protects everything else. Your logins, your credit cards, secure notes, or the office Wi-Fi password. 1Password generates as many strong, unique passwords as you need and securely stores them in an encrypted vault that only you have access to.

29:07I use 1Password, and you should too. 1Password's award-winning password manager is trusted by millions of users and over 100 ,000 businesses from IBM to Slack. It beat out 40 other options to become Wirecutter's top pick for password managers. Plus, regular third-party audits and the industry's largest bug bounty keep 1Password at the forefront of security. Right now, my listeners get a two-week free trial at 1Password.com slash IonAI. That's IonAI all run together, E-Y-E-O-N-A-I. That's two weeks free at 1Password.com slash IonAI. onepassword.com slash IonAI. Yeah. And you said that there's been very strong response.

30:04How big is the... Well, first of all, how do you charge? Yeah, so we have four different tiers. So there's a free tier. So everyone, when they sign up, they get a free tier. They automatically can run a studio for free anytime of the day, right? CPU studio. So they literally could code on the cloud all day long. They can make infinite studios and just they can only run one at a time. So if you want to run multiple, then you're going to use some credits, right? So you have to buy credits. You pay as you go as you need them. We also give you 15 credits, right? But 15 credits means you can basically run about 22 GPU hours per month, which is a lot, right?