Mojo and Building a CUDA Replacement with Chris Lattner

25:30It's the thing that's under service. It's the huge problem. And you were asking about what makes Mojo interesting. The second part, besides powerful compile time metaprogramming, is some basic compiler nervery. And so if you want, we can talk about that. I'll try to keep it high level, but we have a whole new compiler infrastructure that I've been building for many years now called MLIR. And so using a powerful new compiler backend is what enables us to have both really good compile times and things like this. The metaprogramming system really builds on this, but then also it allows us to talk to lots of hardware.

26:05And so that's something that's very different. And there aren't other languages that are widely used that actually do this. And so this is another reason we had to build Mojo. Yeah, so I would like to nerd out on this because I love to geek on this stuff. And I think, once again, you have such a great background for diving into this. So maybe first, let's start with very high level. I think a lot of stuff today is built on LLVM, which you obviously are very familiar with. Maybe give people who aren't aware the very high level of like, what does that structure look like so that we can give them the context of what is this new structure with MLIR bring to the table or how does it shift that world?

26:44Yeah. So let me go back in time. 25 years ago, I was a university student at the University of Illinois, right? And at the time, gosh, 25 years ago, Java was the cool thing. And so virtual machines were the hot new technology. And they weren't new, but it was taking over the world. And everybody assumed that Java would be the thing that unites all of compute. Now, Java is a beautiful system. And the Java virtual machine, beautiful system for a lot of different reasons. But it struggled because it was really designed for just-in-time compilation. And so it was really about, okay, you load an app off the internet, you jit it.

27:20and then you execute the code. And so for larger scale applications, that having to compile it all before you started running it was actually a bottleneck. And so LLVM came onto the scene and our initial idea, which lasted about three microseconds, by the way, was to make a better Java compiler thing that didn't have this just-in-time component that was as heavy weight as what Java did. Now, when we played it forward, the way I architected and built LLVM out was to make it a very generic and composable ecosystem. And so it's a modular design, the way that you can put together the pieces within the compiler designed to be composable.

27:55And so what LVM evolved into is it evolved into the universal connector for CPUs, basically. And so it's design, it has one, what's called an IR, the intermediate representation. And if you look at it, it's well documented, you can go read the spec if you want. if you look at it it's like a compiler person's take on c and so you've got integers and pointers and floats and structs and you've got simd vectors and so you have some of the basics there but it's really c and the way that lvm and then the family of languages around it evolved is they said okay well if you take something like c++ or you take rust or you take swift or you take one of these other higher level languages it's a somebody else's problem to figure out how to map a class down to basically C.

28:37And in the case of C++, okay, figured that out. Like what Clang does, and I built the Clang C++ compiler along with a big team, is it says, okay, well, there's VTables. And so we can lower a VTable into your code and LLVM can represent that. And that was fine. Now, more modern languages like Swift and Rust and things like this really benefit from higher level optimizations. And so you really want to be able to do devirtualization. You really want to do monomorphization of templates and things like this. They don't want to do that on a syntax tree. And so LVM is still an amazing thing. It's widely used for lots of different reasons.

29:11But as languages evolved, there became a need for higher level IRs and higher level representations of the program. And so Swift has its thing called SIL. Rust has its thing. Everybody started building their own thing. And so what this drove is this drove a couple of exciting opportunities because languages evolved and that was very powerful and yay, go technology. But it also drove fragmentation. And so the cool thing about LLVM is that it's united so much energy, like the chip providers could say, hey, if I just add hardware, if I add support for my chip to LLVM, then I get all software, I get, you know, Linux, I get a web browser, I get like all this stuff just by adding it back into LLVM, I get Swift, I get Rust, I get Julia, I get all these beautiful things that come just by adding LLVM support.

29:55that's cool but now and what that did was that enabled a factorization of industry effort and so people weren't re-implementing the same stuff 50 times over they could get high roi high impact for their work and they got this massive software ecosystem now as languages evolved you got this fragmentation and so the swift people have their thing the rust people have their thing the julia people have their thing etc well guess what ai people didn't notice and so what happened is when AI came on the scene, they started building all their own stuff. And so their own intermediate representations, you got TensorFlow graphs, and then TensorFlow had TensorFlow Lite, and then you had TorchScript, and you had Glow, and then you had XLA, and then you had like this explosion of compilers, Onyx, like all these things got built.

30:39And as all these things got built, they, you know, what a compiler people do, they go and build a compiler, right? Well, now what you're doing is you're fragmenting the talent ecosystem. And I love compiler engineers, I know many of them, They're lovely people, but there's not very many of them. And so if you take a scarce resource... You might say you know most of them. I probably know most of them, like over half, probably, the people who represent as Compiler Nerd. And so you take a scarce resource and you divide it and you get this really unfortunate problem where many of these technologies are made by really well-meaning, good people, and they're very talented.

31:13But there's like two people on every project. And so they're well-meaning, they're good, they're very talented. But if you only have two people, there's only so much you can get done. And so, and they're trying to tackle these new spaces like heterogeneous compute, GPUs, crazy custom accelerators, like all this stuff. And it became very untenable. And so MLIR was a project that I built, again, out of need. And when I was at Google, and at Google, my day job was I was responsible for getting the Google TPU to work, get the software ecosystem to scale, get it launched in cloud, get TensorFlow to talk to it, get it integrated into PyTorch.

31:48these very big, audacious problems. But I was also in charge of all the CPU, GPU performance. Google also had many unannounced ASICs that were both data center and edge and many other things. And so all of these things had different compilers that are all built for their one little niche. And so what I realized is I realized that the LLVM architecture didn't make any sense. You can't say, hey, Google TPU, which has a huge matrix multiplier as a primitive. It's not C. like that is not the right abstraction and what i learned from swift and from many other systems since then was okay well there isn't one right answer what actually you need is you need the ability to build domain specific compilers and build them very efficiently and we need to refactor the ecosystem we need to get compiler engineers to talk to each other again we need to get it so that we get compounding interest out of our investment and so what mlir is fundamentally is it's a way of writing domain specific compilers with very high leverage so you don't have to re-implement all the basic compiler stuff like a constant folding pass, you can instead really focus on your domain and get great leverage out of that.

32:55And so this is something that has kind of taken over our ecosystem. I mean, it's widely used. There's lots of AI things that use this, but also your last show that I heard quantum computing massively into MLAR because it's a very important domain. I've used it for chip design. And so making it so you can actually synthesize Verilog and then synthesize hardware underneath that, that's very different. That's not C. And so like these kinds of applications were really enabled by this. Yeah, no, I love that. So just to flesh back out or replay this a little bit, LLVM and particularly the LLVM IR did was kind of create a choke point for when things looked mostly C-like and were mostly compiling to target CPUs.

33:36Everyone could, if they had any language, they could build a little layer that compiled down to this C-like IR. And on the flip side, if they're building new hardware, They just need to make sure that there's a plugin that knows how to compile from that IR to their chips, machine language, what have you. And the breaking point was, oh, new hardware, heterogeneous computing, all these different things. Like that IR does not sufficiently express what we want to express. It doesn't allow people to do it. So you're kind of in some ways, if I understand it, MLIR is almost a higher level IR that allows you to express many more concepts than could be expressed within C and then also do build sort of reusable abstractions at that level, optimizations, and then there's another layer for people to target.

34:21Now, two questions that I have coming into this. Actually, I'm going to start with the sort of downwards path. So from MLIR, you have this intermediate representation. You're able to do whatever you're doing. What is below that? Does that get lowered or is that able to then go straight to machine language? Like, what does that path look like? So it depends on the compiler you're building, but typically you end up with LLVM. Okay. So this is kind of another layer on top, essentially. Yeah. Which makes sense. I mean, compilers are probably the deepest abstractions known to man. That's right. Right.

34:50So if you're building an AI compiler, which is where I'll focus, but there's also lots of other cool stuff you can do. But if you just look at AI compilers, typically you're starting with tensors. And tensors are gigantic multidimensional arrays. And then you need to lower it down into, I have a 4x4 matrix multiplication. Things like this. Or I have SIMD operations on a CPU. Things like this. And so that mapping process is very complicated. But once you get down there, then LVM is amazing. It's really good at doing register allocation and scheduling and the core things and cogeneration. And so it's quite good at that.

35:21And so it's about how do we solve these higher level things? Okay, so that's awesome. And I love that. The next question is kind of at that MLIR layer, you highlighted that you're trying to make it useful, not just for GPU programming, but for kind of programming any number of these sort of accelerator type heterogeneous computing problems. So what is the abstraction that this MLIR operates at? Like, what is it that makes that work? So MLR does not. So this is where Mojo comes in. But let me let me express why this is. So MLR solves a compiler construction problem. And so it's really good at making it so compiler experts can build a new compiler quickly, but it does not solve design, does not solve any specific abstraction problem you might have.

36:10Let me just give you an example of this. MLR has many, many, many different ways to map tensors onto hardware. And if you go check out MLIR.LVM.org, you know, nice plug, you can go read about these things. None of them actually work very well. And so you need, just because something can represent different levels of abstraction. And by the way, the ML and MLIR doesn't stand for machine learning. The ML stands for multi-layer. And so it's about progressive lowering. It's about representing things at the right abstraction level to do the optimizations you want to do. I sometimes joke that as a compiler engineer, That which you can represent, you can transform.

36:48And so a lot of the power in compilers is getting things at the right level of abstraction so you can do things with them. But you still have a burden of having to decide what those abstractions are. And MLR really doesn't help you with that. And so what's been happening is the industry at large has been stabbing around the dark, and there's a million different projects, and none of them have really gotten traction. And so this is why Mojo has to come into this world. Because, first of all, there aren't languages, really. Actually, I'm going to blow your mind in a second and tell you the other language that targets MLR.

37:17But generally, people have been integrating these things and trying to get them to talk to PyTorch or TensorFlow or things like this. And so that's been a huge challenge on its own. Does that make sense? I think so. So I guess then a couple of different things that I'd wonder is, so multilayer is cool. Are those layers well-defined right now, or is that also flexible? They're completely flexible. And so MLR has a concept of a dialect. And so you can define a dialect for your hardware, for LVM itself, for your programming language, for quantum computing. It's domain specific and you get to own the domain.

37:52This is what I'm saying where MLR is very powerful and very awesome, but the power gets reattributed back up to the person holding the hammer. Because now you can go, it's very powerful hammer. In some ways, right? It's saying, for example, I have a new piece of heterogeneous hardware, a new accelerator. I can build a domain within this that understands how to translate from some of the higher level MLIR into the way that this is going to work optimally on my device. And now it can get lowered into LLVM IR and go. Exactly. And so now here's the thing, right? So Mojo exists because the entire industry was not doing it right.

38:29Right. You need a proof point. You need an example for people to see and follow. For years, people have been successfully taking MLIR and they've been integrating into existing systems and upgrading things and it's had huge impact. I'm very, very, very proud of MLIR and it's got a massive ecosystem and lots of different people use it. It's very, very exciting. But there is no language or there is no really defined programming language that uses it. There's a lot of domain-specific languages or EDSLs or things like this, but there was nobody that was actually taking this and using it in a way that expresses full power of generic heterogeneous hardware back to programmers.

39:02It was more about, like, try to get PyTorch to go fast. And some of those sort of worked, but most of them did not, right? Now, I have to be careful because Mojo is very uniquely designed for MLIR, but there's one very brand new, very exciting new programming language that now directly goes to MLIR as well. This is a new LLVM project that is just graduating to the next level of maturity, the LLVM Fortran compiler. Interesting. Okay. Okay. And so now you brought up earlier, let me connect the dots here, right? Well, so in this case, it's kind of an accident that the folks working on Fortran wanted to build a new compiler for Fortran because all the old ones were the wrong thing for a variety of reasons that I'm not the expert on.

39:42But now, to your point, Fortran has parallel arrays, and it has higher-level abstractions that have to be lowered. And so they said, hey, well, this MLR thing's great because it allows me to represent my Fortran domain-specific constructs. By the way, modern Fortran has objects, and it has all kinds of crazy systems. It's come a long way since I was working on it in the early 2000s, yes. And so it's easy to make fun of Fortran because it's from the 1960s or 50s or something like that. But actually, Fortran has evolved into a very modern language with a massive community of its own. Maybe not as many web apps built in Fortran.

40:13But if you look at this, it's the power of MLR that allows people to tackle these problems and solve them in a great way. Now what Mojo does, so coming back to what, you know, now that you get some idea of the wackiness of MLIR and both the power, but then also the curse that comes with having to make design decisions, right? What Mojo says, it says, hey, well, the problem with a lot of these systems and compilers and these PyTorch thingies and like all the different chip people building stuff is that they are functionally taking power away from the programmer and locking it up inside the compiler.

40:48and so what mojo does is it says hey you know what's important library design let's take tensor lowering let's take all this magic out of the magic compiler that a few people are building and let's put it back in user space let's put it into the mojo code let's enable people to tackle tensor cores or their hardware or whatever else it is and let's give that power back to the programmer and let's teach a new generation of people how to actually deal with all this cool stuff. And the way that we do that, in my opinion, is through software engineering. It's one of these old things that is not magic, but it's actually a lot of hard work in design.

41:27It's about building libraries, building abstractions, building ecosystems that are layered the correct way and putting that in a way that's accessible so you don't have to be the hardware expert. You don't have to be the compiler engineer. I love compiler engineers, by the way, but it's like redemocratizing all this technology instead of trying to lock it up into libraries written in an assembly or into compilers themselves. And so that's what makes Mojo very exciting. And so that's why having metaprogramming is so important to me, is because now you can allow people to write just code that has the power of traditional compilers.

41:58And that is, I think, very profound. And we're using it to go solve problems for AI and GPUs and things like this. But I think this is a very powerful thing that as time plays out, because these things take time, I think the world will discover and be able to use in new ways. This is fascinating. And this is a domain that I like to bring up every now and then and geek out on, which is just like, I feel like DSLs in particular are massively underutilized. They're incredibly powerful. There's some interesting examples that I, so for a long time, I was very much in the web space and I saw Babel arise and Babel is essentially a user space compiler that allows you to write language features, DSLs, whatever, or do other different things that compile all to JavaScript.

42:41And this was utilized for things like building out JSX for React and things like that. And I looked at that, I'm like, oh, this is going to create a thousand flowers. And like, it basically stopped with JSX. Like, there's not very many people writing lots of domain specific language extensions there. It's still incredibly useful tooling. It lets you do, you know, a bunch of different things. But I'm kind of curious how, like, it feels like language design which is what we're talking about here empowering people to build languages on top of languages another another world that did a lot of this was like the ruby world was very into like dsls and build dsls and things like this yep like that's a hard problem there aren't very many people who are good at language design either so how do you make those flowers bloom so when you talk about dsls so let me also give another plug so i'm writing this a series of blog posts called Democratizing AI Compute.

43:35And so I go through what went wrong with a whole bunch of different systems, including OpenCL and AI compilers and CUDA and many other things in the system. And the one that's publishing this week is about Triton Lang and domain-specific languages that are embedded into Python. And so what is a domain-specific language? Well, they're actually all around us, right? There are things like SQL or regular expressions or even And HTML is a domain-specific language, right? And so these are quite powerful. There's a subset called an embedded domain-specific language, which is where you don't invent syntax, but you hack the compiler, you hack the Python interpreter, you hack it so that it looks like Python, but it's not.

44:16And so the blog post lays out, these are both very powerful and very exciting and very nice, but they're also very cursed because it looks like Python, but it's not a language. And so now you're beholden to some weird compiler, some weird system that is now, again, it's using Python to hack it so they don't have to do all the work of building a language, but you don't get the same quality result. Mojo is completely different. So Mojo, what we're doing is we're saying Mojo is a language. It has an LSP. It has a debugger. It has a code formatter. It has a compiler, obviously. It is a language. It is a full-fledged, the hard thing to do.

44:51So it's a different energy state than a DSL. But what it enables you to do is it enables you to build very powerful libraries. And so the canonical example of this that is a language and enables you to build powerful libraries is Python, right? And there's a lot of very powerful Python libraries. You can look at PyTorch or TensorFlow or things like this in the AI space, but gajillions of other ones, right? And so we're in that space, which is enable people to build powerful libraries and then allow even more people to build on top of them. And to me, that's really what the sweet spot is, is I shouldn't have to build an EDSL.

45:22I shouldn't have to go hack a compiler. I should just be able to write a struct and use types. And then other people who don't want to know how it works can build on top of it. And this is the miracle of software engineering, which is basically it's collaboration, right? It's about ecosystems. It's about people working together. It's about the power of, again, having two people is amazing. But having two people build something that's used by 20 people, that's used by 200 people, that's used by 2 million people, right? I mean, this is where software leaps get made. And so what my goal is with Mojo and also with Max Together is to make it so you have the ability to unite the ecosystem.

46:02You need all these people that have different levels of expertise and get them to work together. Because I'll tell you, I don't know the latest Blackwell, TensorFlow, Goofy layout nonsense because it's not compatible with Hopper and blah, blah, blah, blah, blah. Like all that kind of stuff. Other people do. So once they build the library, I can build on top of it. that's power right and i don't want to have to like switch between different systems every every second thursday because i have a slightly different use case so let's follow that thread a little bit and talk about what does the mojo community look like today who's developing it what's governance like what does the library community look like like what's going on there yeah so modular is driving mojo and so we're we're funding it we're paying for it we're building all stuff out we are open sourcing it over time and so we have an open source standard library and The standard library is about half of Mojo, by the way.

46:51I didn't dive into it, but like int and float are not part of the compiler. They're part of the library. Again, this is build powerful, composable orthogonal language features so you can put all of it into library. So yeah, int is... So what is the language feature that int and float are built on top of? They're built on top of, they're all structs. And so like int is a struct, struct int, and then as a dunder add method, just like you do in Python. and then you have the ability to use inline MLIR. And so you basically say, okay, well, it's turtles all the way down, zero cost abstractions, turtles all the way down until you bottom out at the low level compiler guts that you're talking to.

47:28Fascinating. So you built Int in the library using inline MLIR to define how the operations work. That's exactly right. And it's all open source. You can go check it out. It's super cool. And so our community has, we have discourse for an online forum. And so there's a bunch of discussion there. We have Discord, which is for the chat stuff. And so we have a bunch of folks there. I have no idea. It's like 20 ,000 people in the Discord, something like that. And so they're all kind of doing different things. We've made the commitment. We're open sourcing the entire Mojo implementation, but we're doing that in steps.

47:59And so Mojo is still being developed. I have some scars and battle scars from Swift where we spent way too much time arguing with people online about things. And so the short version of evolution of Swift is it was secret for four years, and then it got announced and launched. And then it was proprietary for a year, but public. And then it was open governance, open everything since 2015-ish, something like that. And I think that went too fast. Swift, it was great to have community. I love open source, by the way. I've written a few million lines of code that's open source that others that are not.

48:35But it's about the right time and the right place. And having a small, coherent team driving the core architecture, I think, is extremely important for the early phase of a project. And so we're opening things up over time. And we've made a public commitment. Well, I think we'll exceed expectations there. Got it. Well, and as you stated, you're keeping the core language, which is what you are driving with that dedicated team, small, a huge amount is available for anyone who wants to by writing libraries. Yep, that's right. And Modular keeps open sourcing more stuff over time. And the other thing I'll say is that Macs, if you zoom out from Mojo, so Mojo allows you to program CPUs and GPUs and make stuff go fast.

49:11And this is very powerful and integrate with C and Rust and all this ecosystem, right? Max is that next level up. It's the AI solution. And so it enables you to write models. And so we have our own kind of, it's in the space of PyTorch where you can now define your own models and actually build on top of this. And so now you don't have to know how a matrix multiplication works or something like that. You just say, give me one. And you get very fancy compiler techniques that do automatic code fusion and things like this. And so you get very high performance from the max, it's called a graph compiler, right?

49:43And so this then takes this and the way that works and the way that's really novel is that it builds on top of the power of Mojo. And so all this metaprogramming stuff, the ability to have the MLIR, like all of these very nerdy compilery things enable us to build the next level of system that's way more powerful. And it's also an open box. And so just now we're now documenting how to do this stuff. And so we're teaching people how to program GPUs. Well, this is a huge thing because the code you write in Mojo with max and together, yeah, you can use it to solve AI, which is awesome and very, very exciting for lots of different reasons.

50:15But we're really about enabling heterogeneous compute, enabling people to use the accelerator, enabling people to build and scale across hardware and all this stuff is free. And so this is a really big deal. And while some of these technologies are young, like they're very, very powerful. And so I love to see, to your point about community, like what people are doing with this and can do with this. So that's, and I talked with people, last week was talking with a whole bunch of people that are talking about robotics and all the applications. And there's apparently this Python ROS thingy and moving chunks of that to Mojo would make it so much faster and better and solve all these problems.

50:51I'm like, wow, I know nothing about that. That sounds really cool. So that's what I love to see. Yeah. No, I love that. Well, I just realized, I mean, I feel like we could keep going for hours, but we are getting close to the end of our time. Is there anything that we haven't talked about yet that you think would be really important to leave folks with? Yeah, I mean, the thing I'd love to say is that I get very frustrated with the industry right now because if you look at the powers that be out there, you've got these massive companies doing AI, right? You've got these very well-funded AI research labs that are making the news all of the time, right?

51:28And you have all of this just investment in AI and it's all, you have the hardware companies like NVIDIA that are saying like, okay, well, Cuda, Cuda, Cuda, right? And so what I see happening out in the world is I see all these people that are telling you, telling us all, the AI is too hard, give up, air quotes, only we can do it, right? And whether it be the big LLM company of the day that's saying like, just use our endpoint or it's the hardware company saying, you have to buy all of our hardware and only from us or the big tech company that's saying like, hey, you know, all this stuff is too complicated, just use our cloud service or something, right?

52:04But it's all a lie. The stuff is complicated because the software is complicated and because this was all cobbled together and it doesn't make any sense. But the technology is not that complicated. What I strongly believe in is as we democratize this stuff, as we make it actually easy, if we make it so that people have power back over the software, suddenly we can have another wave of innovation. And we can take back power from all these like overfunded, overly powerful ecosystem players. And we as a community can rise up and we can achieve a lot of the stuff that they tell us is impossible. Because, I mean, I am empowered when I talk to new college grads, right?

52:44And so I'm an old dog at this point. New college grads know all of this stuff. They know how models work. They know how sometimes GPUs work. They know all this stuff. But the problem is that we have this fragmented talent ecosystem. And what I really believe in is if you get the people able to work together and collaborate and build stuff in the open together, like the whole world will continue to change in a much more powerful and positive way than, okay, please just use our endpoint because you're not smart enough to know how AI works, which I don't believe. So this is a big part of the mission.

53:15Also, as a nerd, right, all the hardware out there is so exciting. It's all lacking software. And so we're trying to unlock that. This will be a big part of our storyline, particularly as we get into late this year. but I think that's a huge opportunity. And the way I look at hardware today is that everybody's crying about NVIDIA GPUs and the prices and all this kind of stuff. But when I look ahead five years, 10 years, I look into the future, I know deep down in my soul that hardware is going to be even more weird than it is today, right? Physics, we're not in the age of Moore's law. Innovation is not dead.

53:48New algorithms, new research, all this stuff's going to continue to happen. And so what we need is the ability to scale into this. And this is what I think we're trying to do. And very excited about that. No, I 100 % agree. We are in an age of increasing weirdness and heterogeneity in the hardware world. It's also a golden age of hardware innovation. There's so much going on because we're suddenly in a place where you just can't get enough compute. And so everybody's trying to innovate and change the boundaries. And we need programming languages that expose that to us as software developers. Yep.

54:18And programming languages fundamentally unite communities. right and so that's where again what i'd love to see is different people with different backgrounds different perspectives different use cases that can collaborate and solve problems together this is software right this is what's powerful so if you're interested please do check out our webpage we do have a ton of stuff we're open sourcing things all the time we have major new releases coming out regularly there's all kinds of new capabilities please join our community read the blog if you want to learn about the history of ai compute and why the software is so screwed up trust me, I have decided that instead of telling every individual person, I should write it down and scale this a little bit.

54:53And so if you find that, you know, why didn't OpenCL win or something, please, I'm happy to teach you about it. So awesome.