The Vulkan Graphics API with Tom Olson and Ralph Potter

0:00Vulkan is a low-level graphics API designed to provide developers with more direct control over the GPU, reducing overhead and enabling high performance in applications like games, simulations, simulations, and visualizations. It addresses the inefficiencies of older APIs like OpenGL and Direct3D and helps solve issues with cross-platform compatibility. Tom Olson is a distinguished engineer at ARM, and Ralph Potter is the lead chrono-standards engineer at Samsung. Tom and Ralph are also the outgoing and incoming chairs of the Vulkan Working Group. They join the podcast to talk about earlier graphics APIs, what motivated the creation of Vulkan, modern GPUs, and more.

0:43Joe Nash is a developer, educator, and award-winning community builder who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Gary's Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

1:15Welcome to the show. Thank you so much for joining me today. How are you doing? Hi, Joe. Doing good. This is Tom Olson, by the way. Awesome. Perfect. How about you, Ralph? Hi, Joe. Yeah, I'm doing good. Thank you for having us both. Awesome. So to kick us off, you know, I introduced you both there as the chairs of the Chronos Working Group. There's a bit more to both of your stories. Tom, do you want to kick us off by introducing yourself and how you came to be working on Vulcan? Sure. So I work for ARM, which is known as a CPU company, but we do also make GPUs, the Mali GPU family. And I've been a professional graphic standards committee chair for the past 18 years.

1:53It's a bit horrible. I started chairing the OpenGL ES standard, which some folks may have heard of. It's the mobile version of OpenGL. And when the need came out for Vulkan in 2014 or so, I, for my sins, picked up the flag and ran with it and helped to get that effort started. And I've been doing that ever since. I've reached the stage where it's time for me to get out of the way and let younger people take charge. So hence Ralph. Perfect. Speaking of those younger people, Ralph, how about you? Yeah, so I work for Samsung where I'm located in our GPU team, specifically Samsung Mobile, the portion who make kind of mobile phone handsets.

2:37I've been in the Vulcan working group for six or seven years now, five of them representing Samsung. And yeah, I don't have Tom's 18 years of experience of chairing, but I have a little bit and it will be hard to replace Tom, but we will do our best. Of course. Can I point something out? Yeah, please do. Both Ralph and I have been working on standards before our current employers. This is, it's kind of the culture of the group. There's a large number of people involved with creating Vulcan where their involvement persists across multiple employers. It kind of gets in your blood. It's hard to put down and it's a rare skill that companies need.

3:18So they will often hire you to keep doing what you're doing. Fascinating. So actually, yeah, that leads me into, I guess, a topic that I think will be interesting to explore, which is how Vulkan is developed and what Kronos is. But I guess first to set the scene for folks who somehow aren't familiar with Vulkan. Tom, can you tell us briefly what Vulkan is? Sure. So Vulkan is the modern way to program GPUs. In the past, you've heard of APIs like DX9 and OpenGL, and those were kind of graphics APIs, and there was a big magic driver that turned that into GPU commands. With Vulkan, what we've done is remove, it's not really a graphics API, it's an API for controlling and programming a GPU.

4:04All GPUs today, you probably know, are highly programmable. They have multiple execution engines and you can write code for them, but getting it to run and run efficiently and in parallel on the GPU is difficult. So Vulcan's job is to expose that power. That's a really interesting distinction between, you know, it's not a graphics API, it's an API for controlling a GPU. Very, very interesting. So I guess what are the, you know, you mentioned obviously it's easy to control that power. What are some of the advantages of this approach? What problems are you directly looking to solve versus the previous generation?

4:38So in the previous generation, the basic problem is that a GPU's programming model is incredibly different from a CPU programming model, even a CPU cluster programming model. Massively, massively parallel, data parallel typically, and increasingly flexible too. And the basic problem was that the old generation of APIs, OpenGL, et cetera, presented a sort of very nice, convenient, but very CPU-like programming model, where you gave a command, the device did it. You gave another command, the device did it. GPUs don't work like that at all. You queue up massive numbers of commands and you shove them in the driver and the hardware, take them all apart, execute everything.

5:26It's kind of like a data flow paradigm. Everything runs as soon as it can, as long as the hardware understands the dependencies. And so the result is that with the old APIs, you couldn't get the efficiency you wanted because you were working through this very thick abstraction and this very sequential abstraction. You need an API that exposes the massive parallelism of the device. And so Vulkan does that. Specific problems we had in the old days, OpenGL had no real way to make use of multiple CPU cores so that if you were trying to keep the GPU fed with commands, because the GPU is just GPU is a voracious consumer of data and commands.

6:10And you might need multiple threads in order to generate those commands fast enough. You couldn't do it. It wasn't in the programming model. Vulkan solves those problems. So one of the goals of Vulkan, as I understand it, was supposed to be hugely cross-platform and support lots of platforms, which I guess is kind of shown by both of your backgrounds here between ARM and Samsung, obviously focusing on a whole world of devices. Ralph, can you talk about what kind of platforms you're looking to support with Vulkan? Because I understand it's not, I think most people think of like, you know, PCs and the consoles, right?

6:40I understand it's like a whole world of things. Yeah, sure. So definitely it exists on PCs. All of the desktop GPU vendors have Vulkan drivers. It exists on some of the consoles, both handheld and dedicated ones. It is pretty fundamental to Android these days and becoming more important. So the vast majority of mobile phones that you can buy today will support Vulkan. There are some outliers, but the vast majority will, at least in the Android space you will also see it it may not be so obvious but it also exists on other devices as well more embedded devices you know appliance type devices you might find there is Vulkan in there even though it's not obvious to you as a user so the short answer is if it's got a relatively modern GPU there's a good chance that there is a Vulkan driver available somewhere right I think i saw in a talk from siggraph that someone mentioned a coke machine that might be new tom actually that you know example yeah yeah yeah particularly cursed programming environment cool so yeah i think that really sets the foundation for what vulcan is so i guess to i wanted to talk a little bit about the history which obviously you mentioned a bit in your intro tom about you know how it you know around the start of it and transitioning from open gl and open go es to vulcan you mentioned that there was a need for vulcan and that's when you started working on it what was that need can you give us the run-up and the history to vulcan sure so if If you go, boy, it's shocking how long ago this was.

8:16So go back to, say, 2012 to 2014. The dominant graphics APIs of the day were OpenGL and DX11, which was the most modern version of DirectX on Windows. And they both had this problem that they were lovely programming environments. It was very comfortable and easy to move into using them. They worked the way a CPU programmer would expect a graphics API to work. But people had enormous difficulty getting performance out of the devices. So as a result, for example, on consoles, nobody used them at all. Well, they used DX a bit on Xbox, but mostly they just threw them away because they could not get the performance.

9:00And so around that time, you could say it was kind of a revolution, the farmers with the pitchforks. developers were looking for alternatives and you had things emerge like Mantle, which was an AMD proprietary API since AMD had cornered the console market at the time. And it had this property that it exposed the parallelism at the price of not being as nice a programming environment. It's much more painful to use and complex to think about, but it gave you the power. And so people were very excited about that. And there was talk of moving toward it. Microsoft began moving in the same direction with an API called DX12, which I should say DX12 is to DX11 pretty much what Vulkan is to OpenGL.

9:50We kind of divide the world now into modern GPU APIs, which is Vulkan DX12 Metal, and the old ones, which is OpenGL ES, DX9, 10, 11. So these modern APIs were emerging, and we felt that OpenGL was going to be left behind. We could see that it had problems. Well, I would say we were all working, Ralph too, I believe. We were all working in the OpenGL space. I was chairing OpenGL ES. We could see that there was no way we could evolve those APIs in a gradual way to meet the need, to provide the efficiency that developers were just demanding. And so that led to kicking off the effort. We kicked it off in 2014, took us a year and a bit.

10:38We came out in early 2016 with Falcon 1.0. So did that answer your question? It did. And it also answered another question. Obviously, DirectX 12 and Metal, my timeline is a little bit fuzzy, but my understanding is they're kind of all at the same time or the same generation. And so I was going to ask, you know, how did that happen? Why were they all at the same time? But I think you've really filled that in. So I guess one of the things you mentioned there is, you know, this kind of step change in developer experience in terms of, you know, what the developers could expect the API to do for them and how much extra work they had to put in.

11:07How do you go about navigating that in terms of an API design philosophy? Like, you know, you're trying to meet the needs of the users as, you know, graphics programmers, they come in, you know, there was a demand for this level of API, but I imagine you still have a lot of people who are still expecting the affordances of the old API. How do you juggle that? Painfully. It's a constant, I won't say balancing act, but it's a constant debate. To be frank, in Vulkan, as it is today, well, certainly in Vulkan 1.0, we created an API which gave developers what they said they wanted, but was frankly quite difficult to use.

11:44And we made sort of an intellectual commitment. And by the way, when we started this effort, we had massive participation, particularly from game engine companies, Epic, Valve. Well, Valve first and foremost, they were real champions of Vulcan early. But Epic, Unity, all the majors were there. And we had a big fight about, are we going to make concessions to ease of use? Or are we going to say performances first, full stop? And we pretty much said, no, performance is first. We will never sacrifice that. What's happened is that the hardware has gotten easier to use. And so Vulcan has gotten easier to use in parallel.

12:27And modern Vulcan is not nearly as gnarly and, you know, sharp edged as Vulcan was early. Boy, I think I'm rambling a bit here, but I'm trying to make sure I hit the various aspects to this. I would say one way we deal with this problem is by tooling. So a feature of Vulcan that we think is one of the best ideas ever. I wish I could remember who in the group had it. So a feature of Vulcan, since it's dedicated to efficiency before all else, it does not check for errors. When you give a command to a Vulkan function, if the commands you give it are meaningless, the specification says you get undefined behavior, possibly including program termination.

13:12So you make one mistake and it's dead. Driver restarts. What do you do? Well, what we do is Vulkan has a defined interface to shim layers. We call it the layer system. And so when you create a Vulkan, you're a programmer, your program says, I want to use Vulkan, give me a driver, please. You go through this negotiation. You can say, please install the validation layer on top of Vulkan. So if you do that, you get the same interface, all the same functions. But when you call, if you pass garbage information into a Vulkan function, the validation layer checks it before it calls the underlying driver.

13:56And it logs an error if you did something wrong. The validation layers are incredibly powerful and useful. The investment that's gone into them is millions and millions of dollars. It's very complex software. A lot of it paid for by Valve. A lot of other stuff written by members. But it's one of the most important things. So we're trying to provide you with a safe and sane programming environment, but not at the price of slowing down the hardware. So the idea is you develop with validation turned on. When you ship your code, you turn it off, and suddenly everything runs much faster because the driver is not checking any errors itself.

14:37Ralph, what have I forgotten? I kind of rabbit hole done. Now, I think all of that is correct. If I was going to give one quick piece of developer advice, I would say if you're writing a Vulkan application and you are doing it without the validation layers enabled, you are doing it wrong. and you will come to regret it. They are pretty fundamental to that. I also agree with Tom that this balance of usability and the challenge of using the API is a difficult problem. If you go back to our kind of 2016 launch publicity, at the time we said Vulkan is not the API for everybody. I think it has become less thorny to use as time has gone on.

15:27There is still no free lunch. Like it is still definitely harder to get started in Vulkan than it is to get started in OpenGL. There is a higher expectation. Tom said that this is an API to control a GPU. there is a higher expectation that you will understand how a GPU functions than maybe there was in OpenGL. I think once you have that understanding and once you've got over the initial hurdles of how to get started, the fact that it is more predictable, there are less unexpected driver heroics going on. There is a place in which you can say it is more workable, but it requires a certain base level of understanding.

16:10Certainly the barrier to entry is higher. That's kind of undeniable. It's kind of intrinsic to what we built. It's a really interesting point about if you understand how GPU works, it's easier to use. I feel like in recent years, there's been a lot more general awareness of how GPUs work from base-level programmers because of general purpose GPU, and obviously ML and AI is accelerating people's usage of GPUs. Do you think that understanding is becoming more widespread in the developer community and making it easier for developers at large to use Vulkan or yeah I don't know I think well it's hard to answer that question without assigning like where we are today versus where we are with the old API there is a lot of information out there nowadays you know there's been a lot of presentations a lot of talks a lot of documentation from GPU vendors that will tell you nowadays how things work in the OpenGL three days.

17:07I'm not sure how much of the same information was out there for the general public to consume in the first place. That makes sense and goes along with Tom's point about, you know, because the graphics cards are easier to use, Vulkan is therefore easier to use, absolutely. So you mentioned throughout the explanation, you know, you mentioned Valve and you mentioned members and sponsors and, you know, we had the introductions both to you about working for members of the group. So I think it'd be great to talk back now about you know what the chronos group is how it's organized and how vulcan is developed would you like to kick off with i guess start with you know what is chronos in the working group sure so chronos is well it's an international consortium and standards body with the mission statement of connecting software to hardware and so they generally most of the products they create though not all are interfaces between some gnarly piece of hardware like a video accelerator a computer vision accelerator, a graphics accelerator, and applications.

18:05And their standards kind of range in approach from being hard over in the direction of developer-friendly and relatively easy to use to things like Vulkan, which are, you know, here be dragons, but you'll get power if you use them. It's got about 100, 120 members-ish today, I think. And it includes You know, Samsung, ARM, Intel, AMD, Silicon companies. It includes game engine companies, Valve, Unity, Epic, several others. It includes software consultancies, LunarG, who create the SDK and the validation layers for us. There are other kinds of people involved. It's a wide variety. The thing that ties us together is that there is an IP agreement, and this is very important.

18:56You don't want to read the legalese. But in a nutshell, we agree that if you have patents that are necessary to implement Vulcan or any other of our ratified standards, you agree to license them to implementers of Vulcan at no cost if they're necessary. Patents on techniques for implementing Vulcan, like particular circuits for this or that, you can own and you can enforce the patents. but something that the standard itself necessarily infringes, you have to license. Or you can withdraw from, you know, there's a way to withdraw yourself, but it's kind of suicidal. So it's very rare to do that.

19:36So that's Kronos at a high level. There's a board, there's all this infrastructure. Then there's the Vulcan Working Group, which has the members that I said on a typical call, We have maybe 40 people from maybe 20 companies participating. We do design work on new functionality. And we do this maintenance call, which we did this morning, where we go through and fix the corner cases and answer developer complaints. We have a presence on GitHub and anybody in the world can come in and say, this doesn't seem to work the way I thought it did. Is the spec wrong or what? And we will jump on that and answer that.

20:16And often it results in spec clarifications. We do other things. We make a conformance test so that if you're implementing Vulkan and the spec says, though, you know, implementations must do this, we make sure they do. And that's our biggest expense. We spend about a half million a year, more than that, actually, writing those tests. We have software contractors who do that for us. Then there's the SDK and the tooling. there's a compiler that is used for as i said gpus are programmable you've programmed them in special languages and there's a compiler for that those are all things that we maintain as part of this effort okay that opens up so many questions i'm gonna start with the conformance test so you mentioned if there's an implementation you will test it that's responsibility you take so when you say an implementation what does that mean and i guess my question that came from that is like if someone goes and starts a new welcome implementation just like a random person and And then it says, cool, do you have to test it?

21:15Is that how that works? So here's how it works. Vulcan is a trademark of the Kronos group. And if you want to use that trademark, you have to have permission. And the Kronos groups on the website, the guidelines say you can use it for conformant implementations. And there's some weasel words about, you know, if it's in development and it's not certified yet. But basically, when you've got your thing done, and sorry, let me back up and say, typically, a Vulkan implementation is a device driver. It comes from a GPU vendor and you get it and install it on your machine. Because you have the right GPU, you install their driver.

21:53We have the ability, this is built into the infrastructure, that if you have two separate graphics cards in your machine and two GPUs from two different vendors, you can put in drivers for both of them and it'll work. And your application will have to choose when it's setting up, starting to use Vulkan, say, well, which device do I want to run on? But generally, a Vulkan implementation is a device driver. There's a lovely software implementation out there called LavaPipe. And if you are experimenting and learning and having trouble or you just don't want to install a device driver, you can use LavaPipe.

22:29It's quite efficient, actually. Sorry, I got sidetracked. No, absolutely. That's perfect. Yeah, LavaPipe sounds really cool. And I think that ultimately answers my question. So sorry, I'm now going backwards through your answer to how the Kronos worked. So when it comes to participating in the working group, you know, you work at Arm, Ralph works at Samsung. What is the, I guess, arrangement there? Do the members give employees over to the working group full time? How does that work? No. So, well, it's up to the member. Members of Kronos are typically companies. There are a small number of individual contributors approved by the board.

23:04But generally, to work in Kronos, your company joins, they pay a fee, it's a couple of tens of thousands a year, and then they have the right to participate. And what that typically means is they tell certain of their employees, part of your job is to go to the meetings and contribute to making Vulcan better and making Vulcan work for the community. So in my case, as chair, it's been a full-time job for me. But that's rare. For most of our members, they're working maybe 10 to 40, 50 % of their time devoted to working on Vulcan. And the rest, they're doing things for their own companies. Participation involves, in the case of Vulcan, two 90-minute calls a week, new tech and old tech.

23:50Plus, we have subgroups. There's a separate group that deals with ray tracing, and they have their own meeting. There's a separate group that deals with machine learning, and they have their own meeting. There is a separate group for dealing with the programming language that is used to program the programmable parts of the GPU. And we have a few others. There's like a marketing committee, et cetera. So you can get as involved as you want. You can't really be effective if you aren't spending at least 10 % of your time on it, because you kind of need to be known, you need to have traction, you need to understand what's going on, and there's a minimum cost to that.

24:29Yeah, that makes total sense. So I guess into the particular, so you've got those meetings, but Vulkan has, like any software project, a cadence of releases and things that get added, and the meetings you're having have to come out. Ralph, as the person who's now responsible for this, can you talk to us about the roadmap and how they're constructed? There's a couple of terms that I think would be useful, because I know there's been some change in how the roadmap and how versions have worked over the years, from core version, the roadmap profiles, to milestones. Can you lay all that out for us and how it works?

24:54Yeah. So the first thing to understand about Vulkan is that we have a core API. It's kind of the default set of things that everybody implements the mandatory requirements, on top of which we have a notion of a thing that we call an extension, which is another package of functionality that GPU vendors or implementers who feel that it's valuable can implement this extension specification. And it's essentially an optional piece of functionality that they might decide that their market and their customers see value in. And the way that we used to do things is that we would release a core version on a pretty regular two-year cadence, which would roll up a certain amount of functionality that had been exposed in extensions, but extensions would flow out throughout the year on a pretty ad hoc basis.

25:56And a few years back, we came to the realization that this was getting extremely difficult to handle as a software developer. We have 11 adopters, I want to say off the top of my head, somewhere around that field. And we had all made different decisions about what we felt were valuable extensions. The extensions themselves contain optional sub functionality in them. And identifying exactly what you could expect as a developer became very difficult. On top of which, we had always taken the view that the core API had to be capable of running on more or less everything. You referred earlier to Tom saying Vulkan on Coke machines.

26:46That was sort of a constraint on the core API. It was from 2016 when we launched Vulkan 1.0, we never raised the minimum specs that you needed to run the core on. and so our only approach was to add more and more extensions. As a route to trying to bring some order to that, our process now is we define a thing that we refer to as a roadmap, which is, again, a collection of extensions and features, but we say for a particular subset of devices, we describe it as immersive graphics devices. You can think mid - to high-end smartphones, desktop PCs, consoles, that everyone will ship devices that fit the requirements of a particular roadmap by a particular point in time or approximately that point in time.

27:38So we hope that brings some cohesion. Those have dates on them. So we've released a roadmap 2024 earlier in the year. I don't think it will be a huge surprise to anyone to say that there will be another one coming. And we have now got into a model where we can plan the rough content of roadmaps many years out in advance, which is also important because hardware roadmaps are amazingly long. and if we need to have a conversation in the working group about we would like us all to support feature x if somebody doesn't have it in hardware we're talking about probably five years to go from a hardware design to an implementation to something that shows up in a product and so we have a roadmap that for a couple of years out and we have more tentative things for as far out as 2030.

28:35And when we get that far out, they're kind of nebulous. Maybe they won't arrive in practice, but there's a structure to it now. And so I think my message would be if you're a developer trying to figure out where we're going, the roadmap tells you directionally where we're going. The core API is supposed to tell you what you can rely on to exist on any device that has updated drivers is kind of how I would categorize it. That hardware view and the amount of time that adds to it is, yeah, that's a really fun constraint for working in this world. So obviously you've just told me there that you do plan out far enough in advance, which leaves me no choice but to ask what's next for the next milestone, which I guess is 2026 is every two years, right?

29:19There is a milestone planned for 26. I believe that we shared some of this at SIGGRAPH. Yeah, I had slides on it, but I can't remember. I am now trying to recall your slides, but I mean, things that I know are on there. We have some work on debugging improvements. I believe there's some work on compute improvements. I believe we've talked about ML work to come. We've got a couple of robustness features. Well, we have this expectation that WebGPU, which is the Web Graphics API, which is vaguely Vulkan-like, but much friendlier, and does a lot more work for you and therefore runs slower. But it is what it is.