Creator of Lua: Scripting, Programming Languages, Predictions | Roberto Ierusalimschy

0:00Roberto Ierusalimschy:It's completely feasible that you won't have programming languages. This is Roberto, the creator of the Lua programming language, and I asked him all about programming language design. C++ I think is a good example of a language that is really, really complex. JavaScript I think is even worse. I always joke that JavaScript, the good part, a very thin book compared to the official JavaScript book. Given how much AI has been progressing, would you still recommend people learn computer science today? If you really think about that, it's really difficult to recommend people learning anything. Here's the full episode.

0:45Roberto Ierusalimschy:In 2023, Lua was the second highest programming language by growth, according to contributors on GitHub. And I saw that it was also used in famous games like World of Warcraft and Roblox. And so I wanted to ask you, what kind of programming language is Lua and what sets it apart? Lua is a language that you usually use together, combine it with another language like C or C++. and then you have an architecture where Lua takes care of the more dynamic part of your program, things that change more frequently, things that are not so resource intensive, the hard parts you have written in C or C++. So it's typically what is called the scripting language.

1:39There is some confusion, I think, between scripting languages and dynamic languages. And people just, I think, consider a dynamic language to be more or less the same thing as a scripting language. And they are not exactly the same thing. So for instance, a big difference between Lua and several other scripting languages is that for scripting, you have two typical uses. You can have the ad that I call who has the main loop. So you can have the program written in C and calling Lua, or you have your program written in Lua calling C. And for several called scripting languages are actually, you don't have to, but it's much more easy to use if you have your program written in the scripting language and the C language is only for your libraries.

2:42And Lua, it's a very good fit when you want the reverse. You want the program written in C and then you have, it calls Lua from time to time. So the point is that Lua, it's very good at this that you have the main loop, the program written in C and then it calls Lua for some tasks or for whatever you want to do. And for instance, in games, this is very important to keep the frame ratio of the game. So for instance, you have the loop. The loop keeps the rhythm of the game and keeps everything. And then at every frame, it calls Lua. Lua updates all characters, all images, everything in the game.

3:30and then it returns to C to do the renderization, etc. So it's a very good fit. But the main point is that people sometimes call this the embedding versus extending. So you can extend the scripting language with C or you can embed the scripting language into C. And so several languages are very good only for extending while Lua is really good for both embedding and extending.

4:05Roberto Ierusalimschy:Why is Lua good for embedding? And Python, it's almost the opposite. Because Lua, since the beginning, we thought about Lua as a library. Since the beginning, Lua was written as a library. So I think that's the main difference. Lua has a standalone program that, I mean, you can call Lua in your console, in your common line. But that program is just a client of the official client of the library following the same API that any other program can use. So this idea of language as a library, I think it's not very common. It's something that is very particular in Lua. When you design a language as a library, what are the unique, I guess, design decisions?

4:55I think that goes in a lot of small and big details. For instance, your idea of what is your global space about scopes of variables, about exception handling, for instance. it's very important that it's very common in Lua that you raise an exception in Lua, but you catch the exception in C. So all these things about how you do exception handling in the language. Another, it's not a big deal, but it's, for instance, most dynamic languages. That's almost a definition of a dynamic language. They have an eval function that you can give a piece of code, and then it executes that code. In Lua, we don't, instead of a value, we have a function that we call load that you give a piece of code and it returns a function, an internal function like it was a Lua function that when you call that function, then you execute the associated code.

6:06It doesn't execute immediately. So you have this clear separation between compiling the Lua code that you have, and you can do that in C. So in C, you can get a piece of Lua code. You can compile it, check if it has errors, etc. Then you call that for, for instance, a different piece, or you can even call several times. You can load functions from Lua into C. You keep them, I mean, in the Lua space, and then you can call that same function again and again. It's not a big difference, of course, because if you have a val, you can evaluate a function declaration and then returns that function. So you can, if you have a val, you can do load.

6:56If you have load, you can do a val. But I think load, it's simpler to use for that kind of thing that is, when you think about libraries, for instance. In Lua, it doesn't have a global state. I think that's a very important difference. When C starts, the first thing it has to do is to create a new Lua state. And then everything you do, you do on that state, and that state is completely independent of everything else. So C can, for instance, create another Lua state, and both states are completely independent, completely, there is no communication between them. And so this, again, shows that, and so, for instance, C can use a state to do a lot of stuff, and then it can close the state, and all memory reused by Lua is released.

7:59Everything that Lua was using is released. For instance, if C doesn't need to use Lua anymore For a problem, you release all resources used by Lua, and your programming C continues, and then later you can again create another state, etc. So, as I said, there are several small, or not so small, decisions in the language, but all the time we think about the language as, is that good for embedding? Is that possible to do embedding and things like that?

8:34Roberto Ierusalimschy:When you think of the programming community, generally, everyone is quite familiar with Python, but not as familiar with Lua. I thought it might be interesting to compare the two languages. So if you compared Lua to Python, what are the pros and cons of each language design? Lua is a language that is intended to be used in this idea of scripting architecture. So Lua, for instance, it doesn't thrive having a lot of different libraries. If you have something that, oh, I want to write a quick program for something, that Python is much better. It has all the libraries you can dream about. It has a lot of libraries built in already into the language so the language is huge is i mean the the the installation it's it's a huge download etc but it it has exactly oh i need that oh it's there i need that it's there and louis almost the opposite some very minimalistic language because they did that in bad lure into your program it doesn't use almost any resources most of the libraries the important libraries that you will use will be provided by the program itself will be the comments like move a character or do some speech or things like that from the engine of the game for instance if you think about games but whatever it is so lua i think that's the the main big difference it's that they have very different goals i saw some benchmarks and i saw that lua is much faster than python Why is that?

10:23Those benchmarks are not exactly, but I think that the main point is exactly, one thing is that Lua does have this focus on performance, again, in the realm of scripting languages, doesn't want to compete with C or C++, but in the realm of dynamic, mostly now it's dynamic language, not scripting languages, we have some focus on performance. So I think this is a first difference. Python is actually much more, I think, it's a decision that they make. Performance is not that important. It's more important to be flexible, to be easy to do whatever you want to do. In Lua, sometimes we do not put some features because we think that there is no way to implement that efficiently.

11:13But also, I think some part of that performance difference comes exactly because of the size of Lua. So most of the virtual machine fits in your cache, for instance. I think there is these two big things. One is that Lua doesn't have so many features. It's not so dynamic as Python. So in Python, there is a lot of interactions because everything can mean something else. In Lua, we are a little more conservative on that side. But I think also this thing of being small also makes it naturally faster.

11:52Roberto Ierusalimschy:So on the distinction between scripting languages and dynamic, or I guess it seems like dynamic language is a superset, scripting language is a subset of that. Am I understanding that JavaScript is a dynamic language, not a scripting language from your perspective? Yes, exactly. Yes, exactly. Because scripting, it came, the original scripting language was Bash, or the shells from Unix, of this idea that there's a language that coordinates other stuff. So, for instance, it can be extending, again, you can use, but this idea that you have two different languages, The shell is only useful because you have a lot of programs written in C that are controlled by shell.

12:43So the scripting language has this very strong idea that you have this idea of a dual language architecture. So scripting is like, it's the name scripting. It means it's like you coordinate or you give a script to be executed by those other things.

13:03Roberto Ierusalimschy:You gave a talk a while ago and someone asked you a question of what books do you recommend for studying programming languages? And you said actually that you enjoyed studying the language design of other programming languages. I like reading books that describe the design of the languages. I think the best books are those written by the author of a language about the design of that language. What book recommendation do you think is best on language design? JavaScript, The Good Parts, for instance. It's kind of old now, but I think it's a very interesting book. Although I always joke that JavaScript, The Good Parts, a very thin book comparative to the official JavaScript books.

13:57One hand of the languages, The Good Parts. But I think that book is really interesting because exactly it discusses the language, it discusses even the bad parts and it focuses on the good parts, but then explains why it's there, why it was made that way, etc. That is a book that I like.

14:19Roberto Ierusalimschy:When we were talking about Lua, it sounds like one of the things that sets Lua apart is the performance and how minimal it is. And when I was reading about Lua, I saw that there's a Lua JIT. How does the Lua JIT work? Lua JIT, the first thing that's completely different project. It doesn't have anything to do with us, but it's a incredible piece of software that is a just-in-time compiler for Lua. That's, and I think exactly one of the reasons it works so well on top of Lua is because of the simplicity of Lua. It's a very regular language. I mean, it has a very, as I said, it doesn't have many exceptions or too many interactions or things that, so it's not that dynamic.

15:12I mean, everything can mean something completely different. So that, I think that gives a very good language for Ajit. But Mike Paul is the name of the guy that made the... I think he's still working on that on the first Logite. It's unbelievable, his work.

15:33Roberto Ierusalimschy:What makes writing a JIT difficult? The first thing that for me, I don't want to get involved with JIT is because it's machine dependent. It's not very productive. I mean, you do a lot of work, and it only works on that architecture. and then, oh, I want to run in another architecture. But Paul, he created a kind of pseudo-assembler that he writes in this assembly, and then he translates that assembler to real machine code of the different machines, so trying to unify the different architectures. So there is a lot of, but I don't like, I think architecture, I like to study, but I don't like to work with, because I think it's very unstable.

16:24Each new version, they change that, or they change that. I mean, they change the ADI for something, and then now the stack has to be aligned in some very particular way. And so, and of course, also to get that performance that he gets, He made what's called a trace compiler. The idea of a trace compiler that instead of getting a function and compiling a function, it works like any JIT that it tries to detect things that are executing frequently. And so it starts what's called a trace. It gets recording everything that the code is doing, including function calls, etc. until it closes the loop. And then it compiles that loop, including function calls, etc., everything in line, it compiles that for that specific, for instance, so that number happens to be an integer, so it compiles f, the number will be an integer again, and there is a lot of checks, just check everything is as assumed, and then it executes the loop.

17:42and then it's very, very, very fast. But anything that is different, for instance, you call again that, you call it frequently with integers. Suddenly you call it with a float. Then at some part of the code, it breaks the condition and you have to return to the interpreter part. And I think that's one of the worst parts is that you have to translate the state that it's all compressed into registers, etc. For instance, you don't even have a call stack because you didn't do the calls, you just inline it. But then, oh, something went wrong. Now you have to continue interpreting. So you have to recreate the stack that you didn't create originally, etc.

18:33So even to think about that, I have headaches.

18:39Roberto Ierusalimschy:standard lua is portable but to execute on a machine it eventually gets converted into machine instructions somewhere where in the stack is it eventually translated to machine instructions louis written in c the interpreter is written in c and then you so you you compile that into machine code the interpreter the lua code then the point when you get some some some i mean the the Lua interpreter code. When you get some Lua code, you do what we call a pre-compilation, that is, we translate to an internal language, and then the interpreter is just a big loop, is a loop with a switch. I mean, that's a very big simplification, but in general terms, a loop with a switch that gets instruction, does a switch, and see, oh, this instruction, The instructions are quite similar to a CPU.

19:40Move that, but then instead of creating, it executes that instruction. Move A to B. So it does move A to B and executes that. And again, repeats, executes the next instruction. So this is how an interpreter works. And basically, so we pre-compile a Lua. There is a compilation, but we compile it for this virtual machine. And then we have this main interpreter loop that people call that just fetch hit instruction. A jump is just a jump. You have an array of bytecodes. A jump will just go to that. You have a counter that tells where you are in this array. A jump, you just update that counter. It goes to another position of that array.

20:33Where you're going to, so it's like, uh, you, you emulate the CPU in software. It's written in C, the interpreter. This main loop is written in C. So if you compile it in Linux, it will run in Linux or in a X 86 architecture. It will run in that architecture. If you compile that in the Mac, it will run in the Mac. It's a C program. Got it.

21:00Roberto Ierusalimschy:Okay. So, and that's the part that's not portable and the C compiler handles that. Yes. Yes. Yes. All portability of Lua comes on top of portability of C. How does JIT compare to ahead of time compilation in terms of performance? Both ahead of time. Any kind of compilation is easily, not easily, but it can be 10 times faster than interpreting or even a hundred times faster than interpreting depending what you are doing. But ten times is a very good figure. Between ahead of time and trace compilation then depends a lot of what you are doing. After trace compilation is particularly good for benchmarks because you repeating it more or less they are very uniform you are doing exactly the same thing again and again and again so then trace compilers shine they i mean this is the best they can do in real programs that's depends a lot but the but but you i wouldn't say there is a clear winner between the two i think depends a lot of the kind of program is it possible to write in a Ahead of time compiler for Lua?

22:24Yes, there are several. None, I mean, I'm not sure if there is anyone on production quality, but for research, et cetera, there are several. I have a student who wrote one, like a master thesis, ridiculous, simple Lua compiler, ahead of time Lua compiler, or something like that, exactly. It was just it gets this up codes and expands into C code and then you send that to a C compiler and then you have a head of time compiler for Lua and it gave like three, five times boosting performance. It's a very, very, as the title said, it's like ridiculous, simple compiler.

23:20Roberto Ierusalimschy:I always hear there's compiled languages and there's dynamic or interpreted languages, I guess. But really that distinction is in the tool chain, not necessarily in the way that the symbols are laid out on the source code. Like I could write a compiler or a program that takes in Python code and then converts it into machine code. And then I would have compiled Python. And you can interpret C code too. you can write an interpreter for C. Yes. But the main difference is exactly what it's easy to do. For instance, as I said, one of the hallmarks of interpreted languages is the eval. So if you want to compile the language and keep eval, you have to have a compiler as a library of your runtime because you may want to compile things during execution.

24:17This is the hallmark of a dynamic language. So you can create code while running code. So that part is, that's why much more often than not, dynamic languages are interpreted. But you can compile, but then sometimes, or some compilers do not handle a val at all. they say you can compile as long as your program doesn't have a vowels so there is some restrictions and as i said you can interpret c code i mean but you be extremely slowly we it doesn't but

25:01Roberto Ierusalimschy:i think one big difference and when i look at uh statically or you know static languages or compiled languages they they seem to have type systems in them or the type annotations what is the role of a type system in the compilation process? Depends a lot on the type system. Several type systems like NC, for instance, are written to be a very important part of the compilation process. So if you have the right type system, it's much, much easier to write a compiler. A trivial example is one, the space for variables, because if you know oh, this is an integer, this is a float, this is a double, you know exactly how many bytes of memory you need to that.

25:52Otherwise, it's a mess. More often than not, you have to keep everything in the heap dynamically allocated, and so huge penalty in performance. And for instance, when you see, as I said, a trivial, you have an addition, A plus B. If you know the type of A, A plus B, you have the type of A, the type of B, you have a compile time, you know, oh, this plus is, I'm adding two integers, I just generate the machine code to add integers, and everything is fine. In a dynamic language, I see a plus, I compile to a, in the virtual language, it's just a piece of plus, but then how do I, at runtime, time you have to check what is A, oh, A is an integer, B is a float, oh, I have to convert, or B is a string, or depending on the language, if you're not doing addition, it can be doing concatenation, or you can just calling someone, and you have to do all that at runtime.

26:57So, types can be very, very important to compile efficiently, but, of course, you have to

27:09that there are several languages now, like TypeScript, that the types are a kind of... You do not guarantee that everything has the types you said they have. So then it's impossible to use them to compile because oh, probably A will be an integer, but if it's not, I mean, if I have an integer register to boot through it, A must be a register or things do not work. But if you have the right type system, they are essential for a good compilation.

27:45Roberto Ierusalimschy:I know some programming languages have type inference. What if you ran type inference and you got an unambiguous set of types and then you use that in compilation? Could you do that for Lua? No. I mean that's not computable but a lot of people try to do type inference for dynamic languages and it's a really hard problem and it's very difficult to do anything useful in terms of performance. What sometimes you get but you have a very restricted... If you write your program in that specific way, so it's like you don't have the type, but you write the program thinking about types. And then if the program is that very particular format, then you can do type inference and everything goes well.

28:48Otherwise, either time inference doesn't work or, I mean, it works, but actually it infers very generic types for everything. And so you cannot take advantage of the types because exactly the dynamic nature of the language. This also happens through a JIT, for instance. You can get much, much better performance if you have a kind of a type system in your mind. Usually we do that and you follow that type rules even without, I mean, the compiler, the language does not impose them on you, but you assume, oh, I'm going to follow those sometime discipline. I'm not going to use the same variable to store integers and strings, et cetera.

29:39And then you can have much better results.

29:43Roberto Ierusalimschy:OpenAI, Anthropic, Cursor, and Vercel all use this product to make their lives better. And the problem it solves is when you're building SaaS or an AI product and you want to sell to other companies, there's all these requirements you need to meet. There's SSO, there's SCIM, there's RBAC, there's audit logs. These are all things that take time to integrate, but aren't the main focus of your app. WorkOS is an API layer that lets you meet all of these requirements in just a few lines of code. So let's say you have a new SaaS product and you want to sell to other companies, WorkOS will solve all of these critical feature gaps for you.

30:20Roberto Ierusalimschy:You can check them out at workos.com to learn more and get started. And I appreciate them for supporting my work and sponsoring this podcast. Earlier, you mentioned that Lua can call C and C could call Lua. and you mentioned, we talked about the compiler and the interpreter and how do these pieces work together in the case where you're chaining different programming languages? Yeah, the main trick, I mean, it's not a trick, the technique for Lua calling C is C pointers, function pointers. There is a pointer to a function. this is part of the official C language so when you start Lua suppose you are in C as I said you create a library you create a state a Lua state and then you can register you send to Lua C pointers function pointers and pointers to functions in your C library associated with names Lua stores that in some data structure okay so it says oh the function sign is associated with that pointer etc so when it's doing the interpretation oh this instruction is called instruction called let's see what it's calling oh it's calling a what is the value of a oh it's a pointer And then it calls that pointer.

31:53And then we call the C function to do that. And when C wants to call Lua, I mean, a Lua function is just a data structure from the point of view of C. It's just an array of bytecodes, of opcodes. And then it calls the Lua interpreter function. please interpret that function for me. So C can call Lua to interpret that function and that function is running and then can call a C function using a pointer and you can have that in the stack at several levels. So we are running a Lua function that calls C and C calls Lua again and Lua calls C you have recursive stuff in that etc. and everything works.