In short
Software Engineering Daily - Episode Notes: The X-Plane Flight Simulator with Ben Supnik
Episode Overview
- Title: The X-Plane Flight Simulator with Ben Supnik
- Host: Kevin Ball (KBall)
- Guest: Ben Supnik, Software Engineer at Laminar Research
- Description: Discusses the engineering and technology behind X-Plane, a popular flight simulator known for its realistic physics engine and extensive aircraft systems.
Key Participants
- Ben Supnik:
- 20 years working on X-Plane
- Background in software engineering and aeronautical interests
- Kevin Ball (KBall):
- Vice President of Engineering at Mento
- Independent coach for engineers and engineering leaders
- Co-founded and served as CTO for multiple companies
Summary of Discussions
Introduction to X-Plane
- Developed by Laminar Research; notable for its:
- First-principles physics engine
- Realistic aircraft systems
- Wide variety of aircraft available
- Unique position as both a flight simulator and a platform for user-generated content.
Ben Supnik's Journey
- Began as an enthusiast and modder of X-Plane.
- Initially aimed to transition to air traffic control, but continued contributing due to ongoing opportunities.
- Worked on various aspects of the simulator, including the rendering engine.
Core Architecture of a Flight Simulator
- Similar to AAA games in terms of required high performance and graphics quality.
- Flight simulation has unique constraints:
- Must maintain continuity for existing content created by users.
- Unlike games, the simulator's success is not solely based on entertainment value.
Physics Engine
- Developed in-house, specifically tailored to simulate aircraft dynamics, unlike general-purpose physics engines (e.g., Havok).
- Realism is critical; used for both training real pilots and for hobbyists.
- Can predict the behavior of new aircraft designs based on input data without prior simulation data.
Aircraft Design Format
- Proprietary file format that includes detailed specifications for airfoils, control surfaces, and engine performance.
- Supports user-generated designs, requiring high fidelity in simulation to reflect real-world flight conditions.
Performance Optimization
- Modern mobile devices are significantly more powerful than earlier computers running X-Plane.
- Physics simulations are optimized to run efficiently on various hardware, including mobile platforms.
Graphics and Backward Compatibility
- Transitioned to a photometric and physically-based rendering pipeline.
- Strives to maintain compatibility with older content while improving visual fidelity.
- Introduces new graphical features without breaking existing user-generated content.
Software Stack
- Primarily built in C++, with a focus on maintaining a monolithic architecture.
- Transition to modern graphics APIs (Vulkan and Metal) required significant structural changes to the software stack.
- Emphasis on long-term maintainability and performance efficiency.
Testing Strategies
- Utilizes unit tests, automated testing scripts, and manual testing to ensure code quality.
- Challenges in testing arise from the nearly infinite combinations of user-generated content.
FAA Certification
- X-Plane is often used as a training tool for pilots but does not directly undergo FAA certification; instead, it serves as a component within larger certified systems.
- Compliance with operational requirements is critical for professional use.
Future Developments
- Upcoming features include improved weather systems, enhanced graphics updates, and network synchronization layers for external visuals.
- Development is driven by user demands and technological advancements.
Key Takeaways
- Unique Engineering: X-Plane's architecture and physics engine are specifically designed for aviation, making it distinct from traditional gaming engines.
- User Community: The extensive modding community influences the development and stability of the simulator, emphasizing the need for backward compatibility.
- Long-term Vision: The team prioritizes long-term maintainability and user satisfaction over short-term gains, enabling a more effective software development process.
- Adaptation to Change: Utilizing modern programming languages and concurrency techniques helps address increasing hardware capabilities and user expectations.
Conclusion The episode provides a deep dive into the engineering challenges and innovations in creating a realistic flight simulator, highlighting Ben Supnik's extensive experience and insights into the development process of X-Plane at Laminar Research.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:00X-Plane is a popular flight simulator developed by Lominer Research. It features a first principles physics engine, realistic aircraft systems, and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight simulator, so we invited Ben Supnick on the show. Ben is a software engineer at Lominar, and he's been working on X-Plane for the past 20 years. He joins the show with Kevin Ball to talk about X-Plane and his career working on the simulator. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders.
0:37He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.
1:05Ben, welcome to the show. Thank you for having me. Yeah, I am so excited to get to dig in. This is an area of software I haven't touched before, so I'm excited to go. But let's start with you. Can you introduce yourself a little bit and share about you and X-Plane and what brought you here? I am a software engineer. I've been working on X-Plane for over two decades, which is like 130 years in software engineering years. If you told me back in 2005, you'll be doing this in 2025, I would have been completely surprised. But I started working on X-Plane back in X-Plane 8. I was actually planning to get out of software engineering.
1:47My plan was to make a total career change and become an air traffic controller. and I had been an X-Plane user, an enthusiast, a kind of a modder. And I called Austin up and said, you know, I'm doing this training to become an air traffic controller. I can work part-time while I'm doing this. How about one last thing before I go? I'm going to design a new Siri file format for you and get you some data. And like, that'll be like a sort of a capstone for all the modding I've done. And 25 years later, here we are. I actually got the air traffic control training. and while I was waiting for the FAA to call me back, there was just more to do on X-Plane and more to do on X-Plane.
2:26So I kind of stuck around. I started working on the rendering engine. And by the time the FAA actually called me, we were knee deep in X-Plane 8 and making progress. And I realized I was on this flight team track. Yeah. I would love to sort of dig in with you a little bit on just even what it looks like to be on a project for 20 years, because that's a pretty unusual experience. But let's start with X-Plane. So how would you describe, what is the core architecture of a flight simulator like this? A flight simulator, probably from a software engineering standpoint, it looks closest to a AAA game.
3:01That's kind of where our DNA comes from. But our business concerns are really, really different. So we're kind of outsiders. If there was a room with all the AAA game people, we'd be standing in the corner, maybe not one of the cool kids because we have the same requirements for high performance, as much graphics as we can possibly give them. Realism, physically based rendering has been huge for us, like the whole games industry. But to a certain extent, games live and die based on being fun. And if you don't do that, you're toast. Most games are going to be toast anyway. If you're going to do a sequel, you might just upgrade the whole engine.
3:37Whereas we are kind of a platform because people make content for the flight simulator. People make aircraft. People modify airports. The modding community is so big for us that it's almost like we're a platform as much as we're an app. So we need to have continuity basically as much as we possibly can, right? And that's where we differ from the games where a game engine programmer might say, well, we've got this new technique. We'll have to modify all the art, and there's these constraints. So I get my art director in and say, can we do this? And the art director says, all right, it'll cost this much.
4:11and the producer says, well, we can afford it. And then they tell all the artists, go modify these things. Never use the alpha channel. Here's the restrictions. Here's the bugs you have to work around. And that's how AAA games get made. We can't do that. There's aircraft that people have already spent man years on that are already selling. And the stuff is what the stuff is. So we have to ask ourselves, how do we upgrade the engine and still keep this content working? So it's kind of one foot in AAA games and one foot in platforms, which is sort of a unique niche that we sit in. Yeah, that's really interesting.
4:43So before we dive into any of those, are there any things within the core software piece that would be not recognized by somebody coming from a gaming background? I think from a gaming background, it's all gonna look pretty familiar. Our physics engine is completely homegrown. Austin Meyer, our founder, benevolent dictator, the only Laminar employee for, gosh, like at least 15 years, developed the physics engine for himself to simulate aircraft. And his background is actually in aeronautical engineering. So he has this deep understanding of why airplanes fly. So the physics approaches used there are very different from what you might get in a Havoc or some kind of general purpose physics solver.
5:29Those systems tend to be very generalized and they're trying to give you a bunch of very simple Lego bricks to do some physics in you're a game, X-Plane really deeply understands airplane things. We model airfoils, we model wind flows, we break the wings down into elements. So if you came from a game engine, you go, this is the physics engine. And then you probably go, this is not like any physics engine I've ever seen. What's going on here? And it's because it's very domain specific. That makes a ton of sense. Well, and you have a realism requirement there that most games don't have. It doesn't have to just feel real.
6:02It has to accurately represent what somebody is going to experience in one of these planes. In the case of the physics, it actually goes a step further because X-Plane is not only used to simulate aircraft, and that's the lion's share of our use case is people training for real aircraft with X-Plane and people playing being pilots at home. We have gamers. We have a lot of older users who used to be real world pilots and they've lost their medical. You know, if you've got cardiac problems or eyesight problems, you can't legally go fly a single engine plane. And these people discover that they can rebuild what they had in simulators and still get some joy out of flying.
6:40But we also have people who design new planes using X-Plane. And that's unusual even for flight simulators. Electric VTOLs is an area that's had a lot of growth recently because the batteries have gotten good enough. And once your engine is a simple, inexpensive small electric motor and not a complex unreliable gasoline or diesel engine or turbine you can put four engines on you can do all sorts of novel designs and x-plane can give you pretty good predictions of how that's going to fly just by inputting the design we didn't need to know the answer ahead of time and then spit it back to you which is the conventional simulation approach we can actually be predictive and that puts an even higher requirement on the physics because we have to get it right because no one's going to tell us what the answer is.
7:30That's fascinating. So I'm kind of curious then, what does the format for those airplanes look like? Is it similar to what someone is going to see, for example, as they're speccing things out for manufacturing or like what goes into that? The format is proprietary to X-Plane and it's pretty specific to X-Plane's needs. So we have an aircraft format with just, it's really just a giant key value property table full of stuff. and we need airfoils and their locations. We have a separate subfile to describe the airfoil. The airfoil is described as a table because we don't predict what the airfoil will do from first principles.
8:09That requires computational fluid dynamics and that's a whole nother ball of wax. So once your airfoil has been measured, maybe you have a wind tunnel, maybe you did CFD, maybe you just downloaded the answer because there's a lot of airfoils that fortunately are published, but you've said, I'm going to use this airfoil. You can tell X-plane, here's my wings. These are the airfoils. This is the shape. There's a lot of shaping you can do. They can twist. They can be tapered. They can move while you fly. They can be swept. Here's the control surfaces. This is how much of the space you have. And so all of this detailed data really describes specifically how the aircraft operates in aircraft terms.
8:45So you describe flaps, you describe ailerons. And once you have all this data, including, you know, here's your engines, here's your engine specification, X-Plane can take this and we can simulate the vehicle from the ground up. So we're going to compute how much force did this jet engine put out and where on the airframe did it do it? What was the wind coming into the propeller? How was that wind modified by the propellers as it goes through? Was it redirected? Is it going to then hit the wings after? Is it going to hit the tail? Is the wing going to redirect the airflow away from a different part?
9:20And essentially, we have this collection of parts and we try to derive from the parts what is the net effect on the whole. So there's a number of aerodynamic effects that are emergent, and we don't code them in. There's this tendency of aircraft to pull to the side because of the twisting motion of the propeller. And we didn't say this is how much it pulls. We said, what does the air do? And this kind of comes out by itself. That's fascinating. And it's fascinating because, as you highlight, you don't want to do the CFD underneath it because that's super expensive. but you are doing, it sounds like, a fairly low-level simulation and seeing those emergent effects arise out of it.
9:59How do you make that perform it? I mean, you guys run all the way down to running on an iPhone or an Android. What do you do to make that all work? We run on the mobile phones, but it's 2025. The phones are really fast. The phones are much faster than the computers that were running X-Plane when I started, by a lot. Even an Android craplet, like a low-end Android device is still much better than what we had with X-Plane, say, 8, which would have been a single core, two gigahertz Pentium or something. We don't live in the dark ages anymore. The physics actually take a relatively small portion of the performance budget of the aircraft.
10:41And I think the main reason for that is that the actual number of stations you need to simulate the aircraft isn't that bad. If you can do 40 different simulation points on the wings, that's more than enough to capture the individual effects. But 40 is not a big number for a computer, right? Like we kind of think of it as like, oh, that would take me a long time to calculate. But when computer programs are slow, it's usually because the data input stream is really big. There's 10 million of something. So as X-Plane has matured, the physics has started taking a smaller and smaller amount of the CPU budget, and we've much more spent the Moore's Law surplus on graphics because our users have an unquenchable thirst for it to look more like the real world.
11:26There are some physics things we do now that we could not have done in the single core Pentium 4 days. So modern X-Plane has fairly complicated interactions between the different surfaces. When the wind stream is deflected by one part of the aircraft, that carries through to the next. And those propagating effects are kind of a higher, bigger order than individual blade theory. So if you went back and looked at the original iPhone version of X-Plane back on the 3GS, I think it was, which seemed like a pretty nice phone back in the day, but now we would see this as a small computer, or even the original Pentiums, we were doing all the elements, but we were doing no propagation.
12:08And that is a computationally easier problem. But even with propagation, this is still a much more manageable problem than draw everything on the entire earth within a hundred mile view, which there's a lot of stuff there. And users have been always happy to have more. So that's been where we've kind of sunk our dividends. That makes sense. Well, and as you highlight, that is a similarity to gaming, right? You can never get good enough graphics, you can never do it. But with your constraint of, oh, I have to be backwards compatible to all of these old files, how do you take advantage of those Moore's Law dividends in the graphics space?
12:43So there's a couple of things. First of all, it's a window of compatibility. And the window might be bigger or smaller, depending on kind of how much stress we're under with that compatibility. I would say having good separation of concerns has helped with the compatibility. So I try to think of things in terms of what is this content modeling and how are we interpreting it, right? If you say this model is a drawable geometry with a lighting model attached, your compatibility options are zero because it is exact in the spec of what this model is, how it will be drawn. And so you're not allowed to change things.
13:21But if you say this model is a physical material existing in the real world with some properties on the material, now the lighting model is not part of the model, right? And so you're allowed to make the lighting model better. So anytime we can use the real world as a reference for the content, we have the option of doing better. And anytime we've said the content's behavior is exactly specified, we're stuck. So we can do a better job of simulating the aircraft if you tell us what the aircraft is. And if you tell us how the aircraft works, we're stuck. It better work the same because you told us the how.
13:57And in the graphics field, there's a pretty good separation between the kind of what of what's in the world and how it looks. So what we found is that in X.11, we moved to physically-based rendering, and we needed people to specify their materials in a physically-based manner. And we could look at the previous non-physically-based manner and say, this is the least bad approximation we can do. We can say, this color texture looks a lot like an albedo, so that's the best mapping. And if all else fails, let's do the least bad compatibility path we can. But then going from X-Plane 11 to 12, where we went from it's physically based to it's photometric, meaning not only is it a set of math on the lighting that is based on what real-world optics do, but we're going to try to have all the calibrated numbers be in real units.
14:46Like we're going to specify the intensity of light in Candela. We're going to specify the luminance of a glowing thing in nits. When we did that, to the extent that the previous content was still giving us physical materials, giving us albedos, we could interpret those correctly because we weren't changing those rules. So a lot of that content just worked. And we'd have to look and say, what don't we know that we wish we did? And the answer is, well, it's got a self-illuminating texture, and they never told us the luminance. So we're going to say, okay, for all aircraft surfaces, we're going to assume the luminance is 2 ,000 nits for everything in the cockpit.
15:23And that's the least bad compatibility we can do because we know it's an aircraft. We know the pilot can see these things. What's a realistic value they would need for the simulator to function? And you need to see your displays during the day. 2000 nits. It's not perfect, but it's pretty good. And in that way, we can kind of keep stuff working as best we can. And every time we make a change, we have to ask ourselves those. There's one other aspect, which is a lot of AAA games use a lot of artist time to make things work. right? They have kind of a mission, they have a budget. And if the art director says, I want all the rainforest leaves to look like X, it might be the way they do that is they have code.
16:03They might write an algorithm that processes the art, or they might tell the artist, just go do it by hand. They can make that decision. In our case, I try to be conscious of how the dividends in computer graphics are driving up the price of making content. Because when I started as a modder, one person could make an airliner. The entire thing in about six months was half a man year to make an aircraft because there just wasn't that much you could do. The mesh came from the physics model. You couldn't attach custom meshes from an authoring program. You got about 1k of color texture and that's it.
16:39There were no physical materials. The cockpit was a 2D drag and drop thing. It wasn't a 3D interactive cockpit you could click on. There was no code to write. You got the systems in X-Plane. And so there just wasn't that much to do. And that's all that the computers of the day could do. And as the computers get better and our users expect more, one of the problems has been that the authors making airplanes have sort of borne the brunt of this and that the labor to go into the airplane is going up. So when we're going to add things to the feature, the other thing I think about besides how am I going to map this content, the old stuff, is how much labor are we going to bring in?
17:16Are we doing something that's going to be automatic and just works? Or are we going to ask them to go and do a tremendous amount of work and kind of erode the margin and the viability of making this content at all? So for example, we added proper 3D rain to the windscreen in X-Plane 12. And we take the airflow over the aircraft from that CPU physics model. We put it in the texture and send it to the GPU. And now we can drive the particle system realistically, which is totally fun because you're sitting there on a rainy day and the raindrops are falling down the windshield. And then you crank the engine and they go and they go right off the windshield.
17:55It's very satisfying. And it just works because the GPU has the real shape of the windshield. It has the real flow of the engine and we can just do the simulation. But it's also a feature that's relatively cheap for the authors because the shape of the windshield in 3D was something they had already built for us. They were already building the 3D model and we're reusing it. So this is very cheap. When they do the windshield wipers, we need the path of the windshield wiper in the 2D space of the particle system, which is kind of a hard thing to get because the geometry of the windshield wiper is potentially very weird.
18:31That mechanism could be almost anything that people engineer. And every airplane is a hundreds of million or billion dollar engineering effort. So there's no reason why they wouldn't just build everything bespoke. So we looked at this and said, there isn't a way for us to get this information in real time from the 3D mesh that's even remotely performant. So we bake it. We have the path of the windshield wiper in the 2D space of the particle system as a color gradient where the different grayscale levels tell you these pixels would have been affected by the wiper at this time scale. So now we're starting to make it a problem for the artist because you have to give us this special, magical, exactly right formatted texture to make it easy for us at runtime.
19:14What we did do is we built a script into Blender to produce this automatically. So you could start with your windshield wiper animation, which is something you already built. You do a very small amount of annotation. You say, go bake. And then Blender can sit there and go chunk, chunk, chunk, chunk, chunk, move the wiper, find the pixels, fill them in and export them. So we can still try to protect our authors from having to build something by hand. But in this case, it's going to be done ahead of time. But we're looking at the authoring cost for the entire feature and asking, by making this possible, by making it something our users expect, are we driving up the cost of making add-ons?
19:50Or can we give the authors tools to do this in a reasonably affordable manner? and now their airplanes are better, but they're still able to produce a product whose price point isn't really changing with hopefully a similar amount of man hours. It's fascinating because it's an area I hadn't thought about as much, so much of software, right? The downstream dividend of things getting more powerful is you can write software cheaper because you can skip steps, you can use scripting languages, you can do all of these different things. But in graphics, the expectations have risen faster than the tooling.
20:23That's right. I think it's telling that in software, what have we done with Moore's law? We made our own jobs easier. My father was in hardware for his whole life. And he kind of like scowl at me and say, you kids with your virtual functions, you know, we didn't make the chips faster. So you could just waste all the cycles, right? But the truth is, that was kind of the most useful thing you could do would be to make software cheaper to make or to make software more reliable. I think the dominance of managed languages is telling us something about what people need out of software, right? It's very expensive to have humans write bug-free, unmanaged code.
21:00So maybe we use Moore's Law and not do that. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge and is simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade-in value. Advanced, intuitive, and deployed, that's how they stack. That's technology at Capital One. On that topic, what is the underlying software stack y 'all are using for Xplain? Xplain is coded entirely in C++.
21:37It's pretty much a big monolithic pile of C++. We use a smattering of library code for kind of the things you would expect. like we've got lib pings so we can open pings we've got racknet for a networking sub layer we have things like that but it's very much ours almost surprisingly so and i've had people ask why are you not using sdl2 for your cross-platform layer like why are you doing it yourself and the answer is well the code is 30 years old and there was a time when the thing you're asking us why we didn't use didn't exist so at some point i think we've erred on the side of keeping what we had which was already in production, which is a trade-off.
22:19Maybe our solution isn't as good as a new thing that came out as library code, but also there's often a lot of client code on top of a library, which is why it's often cheaper to rewrite the library or shim it than to go rewrite all the client code. We were OpenGL-based for a very long time because OpenGL was the first hardware API that Austin used for rendering. There was a time when X-Plane was software rasterized on the CPU, But by the time I got there, it was OpenGL and OpenGL runs on multiple platforms. When we wanted to move to Vulkan and Metal, to modern low-level graphics APIs, we had to make a decision about whether we were basically going to just shim Vulkan and Metal under what we already had, or were we going to go rewrite a chunk of the software stack?
23:06And we chose to do the hard thing and rewrite a chunk of our own code. I think if we just wanted to say, hey, it's on Vulkan, that could have been an eight-week Skunkworks project because the API is very narrow and you just shim under it. But all of the architectural contortions that you pick up from being a long-lived OpenGL application would have been still there in X-Plane. We would have had a great tool in Vulkan and gotten none of the benefits because we'd be holding it wrong pretty much always. So instead, I think we spent three or four years of continuous work on this, Sidney Houston and I.
23:41And what we got in return was a real rewrite of the rendering engine itself under modern principles. Vulkan lets you understand the performance cost of everything you do. whereas in OpenGL you make a call some things happen some pixels appear later that's basically the whole contract it's very loosey-goosey and because that abstraction is not well suited to modern hardware and you can't fault SGI they failed to look into crystal ball in 1991 and predict what would happen in 2025 like that's we'll give them a mulligan but because of that OpenGL doesn't have good matching between what's expensive and what you do so realistically what happens in an OpenGL driver is the driver follows along, it squints, it tries to understand what the app is doing.
24:27It does a lot of work to reprocess the information. And at some point, it has to go do a lot of work. And a lot of the times that's at the very first triangle you draw, they go, okay, now we understand the whole problem. Let's go compile some shaders. Let's go make sure these objects are in the right part of VRAM. And all this work that you wish you'd done at load time is getting done at draw time. Our users, especially the professional ones, really want X-Plane to be like a metronome. Rock solid, 60 fips, don't stutter. Stutter-free operation is a requirement from the FAA for professional sims.
25:00So a driver that will sometimes stop and go compile a shader because on this particular hardware that's required because of this state in a way that you couldn't predict, it's really hard to hit that 60 fips limit. But we didn't have any code in X-Plane to go prepare the shaders in the background because there is no prepare shaders operation in OpenGL. You can tell it you're building the shaders, but it's going to half compile them and then wait and then do the other half later when you draw because in OpenGL, the specification for a shader is incomplete. In OpenGL, you give it the source code, but you don't tell it the render target.
Read the full transcript
25:36You don't tell it the fixed function pipeline. It's missing most of the information. So it goes and it does it later when it has the info. So we had to restructure X-Plane and not just switch to Vulkan to get that actual performance win. So we went in and we rewrote a lot of code. It took a while, but in hindsight, it was worth it because we have something much newer, much better matched to the actual hardware and putting new features on top of it has actually been very straightforward. I think that's an example of what happens when you have this very long-term view as well. And you mentioned, I think this might actually be an interesting time to dive into like what goes into maintaining a project for 20 years.
26:16How did you go about that three to four year migration in terms of like planning, phasing, isolating things so that you didn't break stuff that was running? Like, what did that look like? Well, Austin has always said that he wants our time horizon to be infinity when we think about the efficiency of what we do, right? If we say, oh, there's this thing we can do, it's going to be really great in the first three months, but it's going to be inefficient in a 10 year horizon. Like it kills him to think that we're wasting our 10 year time to hit that three year thing. Like he wants us to think about the long game.
26:48And that means we can kind of think about a roadmap of where we want to move the code. So when it came to the Vulcan port, I think it started by trying to understand the technology. We had a pretty good idea of what we weren't happy with. Like we could really see the pain points of what was already there. So we came up with a sort of refactoring roadmap for which pieces we would rebuild and when. and we had to look at how are we going to do this in pieces? How are we going to never have the app so thoroughly broken that we can't test? Because at some point, if you do that, you end up breaking a bunch of things never noticing and then it's really hard to put Humpty Dumpty back together again.
27:27So there's sort of a huge value in never breaking the universe. We also looked at whether we had to stick with OpenGL. One of the sort of the biggest arguments we had was do we have to keep OpenGL working when we ship Vulkan? My answer was yes, and this is what we ended up doing. We shipped Vulkan as a free update to version 11 while keeping OpenGL. And what this got us was the ability to have a very, very long beta. Like, my God, I think that beta might have been nine months, which is atypical for us. But we just needed to have Vulkan out there and find out what it was like on real machines. Because something you get from a shipping product is, you know, what's this like out in the wild?
28:09Because the wild is really different than your lab machines. And in the case of Vulkan, we had no idea what we would find. And we could not have predicted what we would learn from that beta. So having the old system running at the same time meant there was no customer going, I paid$60 for this and it's brick now and I want a refund or I'm going to fight with my credit card company or God knows what. It gave us the time and the space to do the exploration. I think if we'd said, okay, X-Plane 11 is OpenGL, X-Plane 12 is Vulkan, it would have been an easier engineering problem. We would have said, hey, in this new thing, our requirements are simpler because the union of OpenGL and Vulkan is a little awkward.
28:47But on the other hand, God, it better have worked for the first time when you hit go because you just paid for the new version, right? And there's no safety net. You can't go back to OpenGL. And in hindsight, we did not hit that. We needed that nine months of beta to understand how Vulkan performs in the wild. What happens in Vulkan when you're doing memory management and the user launches Chrome and Chrome goes and puts stuff in VRAM, right? Like Vulkan tells you how much video memory you have. And by the way, it might just cut in half and you have to survive that, right? You have to write your own code to cope with that situation, preferably without exploding in a million pieces.
29:23You know, we did not have enough information in the company. We didn't have enough experience with Vulkan before we'd written Vulkan to correctly predict that. So we had to try it in the field. So knowing that, we could then say, okay, which pieces are we going to roll out? And what we found is that there were refactors we could make to our code to use OpenGL in a better way, in ways that probably didn't make a huge difference for an OpenGL driver, but were also fitting the shape of Vulkan. And once we'd made those refactors and shimmed all the OpenGL code into an abstraction layer, then we could rebuild that abstraction layer for Metal on OSX and Vulkan on Linux and Windows.
30:05And then we could kind of hit the lever and see, you know, can we shift all the trains over to the new track? And since we were going to do Metal and Vulkan for Mac and Windows, we knew we would need an abstraction layer no matter what. So having it also handle OpenGL and OpenGL ES in the mobile market, which we thought we'd be on for a while, that was sort of the natural path. And it gave us a way to do the refactor in two pieces. First, kind of making a nice API boundary where previously the OpenGL and client code was completely fused together. And once you've completely cut out that boundary, then you can do a shim and you can switch between them without recompiling the app.
30:43And that kind of makes the refactor tractable. Yeah, no, that technique of introducing a boundary and new abstraction layer, decoupling things, and then introducing these two different ways is that I found that to be highly effective as well. I love what you said about the sort of long term time horizon. I feel like that's something very few of us in the software industry are in companies that support that. But that's incredible. yeah i think we're very lucky to have that freedom and it's one of the ways that we're different than games where often a games time horizon is we're going to do one title and while i have not worked in the games industry my brother does we have people who work for us who have been in games and you sometimes see some pretty scary code because and people write it for good reasons right like we have to ship this we have to ship it on this date and no one's ever going to come and refactor this.
31:31So give me the big rule of duct tape and here we go. And it holds together and it ships. In our case, we know we're going to pay for ripping the duct tape off. And sometimes you end up using it anyway, but we have to think about what it's going to be like in a week, in a month, in a year. Yeah, I wish more software business models took that approach. I think it is definitely a healthier one for the code and for the engineers.
31:57You're a professional software engineer. Vibes won't cut it. Augment Code is the only AI assistant built for real engineering teams. It ingests your entire repo, millions of lines, tens of thousands of files, so every suggestion lands in context and keeps you in flow. Where other tools stall, Augment Code sprints. Unlike Vibe coding tools, Augment Code is built for shipping to production. And you don't have to switch tooling. Keep using VS Code, JetBrains, Android Studio, or even Vim. Don't hire an AI for vibes. Get the agent that knows you and your codebase best. Start your free trial at augmentcode.com.
32:38Feeling the AI anxiety? From questions to job security to cybersecurity and everything in between, it's easy to feel overwhelmed with the rate of AI innovation. Enter ARIA, the enterprise AI orchestration and security platform built to boost your confidence. With ARIA, you don't have
33:16You mentioned testing. Can I ask what your test stack looks like? Our test stack, we have a couple things, and it's never enough. I would always love to have more testing. We have Catch2, which runs unit tests, and we have partial coverage of our lowest stack of software. We've been catching up on test infrastructure because when the company was very small and X-Plane was very small, testing was kind of, let's give it to some users and see what they say. For a very long time, Austin was the only person working on X-Plane. So testing was he He coded it, he tested a little bit, and then he gave it some people and they emailed him at his personal email address if it wasn't right.
33:57And that was everything. We've built that up a bit. So we have catch two for unit tests. We have a Python-based runner of a scripting language because any sufficiently large application has a command shell in it. And X-Plane is no exception. And the command shell, besides providing kind of strange debugging tools and things, also provides a whole bunch of stuff for testing. So we can write these automated scripts. And some of them will do things like we have one that opens up every aircraft with the engines off and does the procedure by invoking commands to start the aircraft, monitors the parameters of the airplanes it starts, and fails the test if something's out of whack.
34:42So if somebody makes a change and now the electrical system is broken and the battery doesn't cause her to be amperage on the bus, test fail. So we have a suite of those. They can also take screenshots so we can get image output. And we have Daniela, our sound engineer, does a lot of web stuff too. And she built this fantastic browser-based diffing program so you can pull up the before and after and spot the difference regions. So we have black box testing. We have unit testing at the low level. we have some integration tests in the middle like you know we have a tool you can run on every airport and it will catch invalid frequency conflicts so when you import new airport data you can run a one-time check i would say the struggle we have is that because x-plane is a platform you can make an almost infinite combination of aircraft and a lot of the bugs that are hard to catch is well it turns out that if you have an airplane with three engines and two are electric One is gasoline and you have two clutches then.
35:40And that particular combination is difficult to cover. And what we don't have is a very, very, very wide suite of aircraft. X-Plane ships with, I want to say, about a dozen or so really carefully built handcrafted aircraft that our team does a great job on. But because they put so much work onto them, you know, I can count them on my fingers. And so the test coverage value is less. And at times we've built simple airframes just to do testing. But I think that's an area where the coverage is poor. We have two QA engineers and they both kind of supervise users of the automated testing resources.
36:18And they also do manual testing. We do have tests where there's a, you know, old school test plan, right? It's a doc with words for what they do. And in some cases you kind of need a human to go do this because automating it would be too complicated. and most importantly, they just sniff out bugs. They're really good at that. We joke about, well, how many milliseconds is it going to take Danny to find the first bug? And they'll come in and they'll kind of shake it and get that spidey sense that a good QA engineer can get. And that helps us kind of catch things that would fall between the cracks that the regular test infrastructure wouldn't catch.
36:53There are people who just seem to have a talent for breaking things and finding where things are broken. It's uncanny when you watch them. It actually brought another thing to mind. You mentioned the many different combinations someone could do in these third-party content, whether it's an aircraft or an airport or something like that. Is there a correctness validation that makes sense in those files or things that are describable in the format but maybe not physically possible? What does that even look like? There are semantic validations that we can do and that we do do. I would say my own work in the scenery code has in the past been pretty slim in terms of semantic validations, mostly due to schedule pressure.
37:35And I've been trying to pay my debts off and add more validation. You know, Austin has a fair number of validations in the physics system because he doesn't want to deal with junk data coming in. But there's also like a gray area between this is clearly wrong. Like you cannot have a stall speed of zero. There's got to be a stall speed versus this is very silly. Like, are you sure you really mean this? And sometimes we say, well, we saw this weird bug and we looked at it and the data was crazy. So we made the validation a little more skeptical. And then we hear back that, no, no, I built an RC plane.
38:10You know, it's one foot long. So all the numbers really are small. This isn't a typo. Like, you know, this is actually how big the prop disc is because it's made of balsa wood. So there's definitely a tension between how much we can validate the model and how open-ended X-Plane can be to very strange stuff. Yeah, for sure. And if people are using it to test out things that they might build, right? Like this can be built but falls off out of the sky is a valid outcome. That's right, yeah. And so flies well is definitely not a validity parameter. On the sort of flies well side, another thing that you kind of briefly mentioned is FAA certification, right?
38:48So X-Plane is not just for hobbyists trying things. It's not just for people manufacturing, but people actually using this software to certify themselves to fly. What's involved in that? What has to be true at the software level to enable that? Surprisingly less than you'd think, but more than you think, not in the areas you might expect. So first of all, they don't necessarily certify software. They certify devices. So we are a software reseller and we sell a professional version of X-Plane to an integrator. And they're going to select the computer hardware, select a version of our software, build a simulated airframe, build the controls, maybe a fiberglass enclosure.
39:33Maybe they're going to buy a motion platform if this is a high-end system. They might have multiple computers networked together running separate displays to get more visuals. And then they're going to take this entire thing to the FAA for certification. And the FAA is less picky than you would expect about the accuracy of the flight dynamics compared to the real aircraft. But they are often very picky about the operation of the simulator. So this one always cracks me up. if your flight control hardware is not working, meaning your joystick is unplugged, your simulator has to not start. It has to like log that as an error and not run.
40:14And the inspectors will come and yank the joystick and out of the USB port. And if we keep flying, they'll fail you. And I've always thought to myself, like, who's going to go in and do flight training time and the joystick isn't working. And they didn't notice like what world do we live in? But that is a requirement like it, because otherwise it's clearly broken. So, you know, We can get tested on that. So the specific regs are quite long. And my experience with the certification process is that it's a little bit ad hoc, that different inspectors have different parts of the regulations that they will check.
40:48And so we sometimes get calls from integrators saying, oh my God, we got called on this thing. It'll be like, we haven't ever seen that before, but sure enough, it's there. So we try to make the most useful, sane, generally highly functional simulator. But I would say that the regulatory structure doesn't match X-Plane's business model because we have a tool that's meant to simulate all airplanes. And the really traditional simulation model is we're Boeing. We built the 737. We spent about a gajillion dollars on the R &D. We're going to get an eight-figure fee every time we sell one. And we're going to make a level D maximally certified 737 simulator that we have made exactly like the real airplane in every way, including using the physical flight deck in that airplane.
41:35And this simulator is a one-off device to simulate this one airplane. And it's going to have, and God knows, seven, eight, nine figure price tag too. And the certification process is kind of aimed at that, I suspect. Like here's a simulator of this thing. So what there isn't is a way to say, here's software that's generally useful. We certify that in the general case, it is a jack of all trades. So we're more of an arms dealer. We're selling shovels to the prospectors in terms of this process. And I don't mind that because dealing with the regulatory requirements is its own complicated thing. So we're happy to do the software engineering, which we're good at, and let somebody else do the certification process.
42:16That makes sense. From a device support standpoint, does that introduce, like these are custom hardware, but underneath it, they're just running devices that you already know how to manage? Or what does that look like? Yeah. So we are on Mac OSX, Linux, and Windows in the desktop space and Android and iOS in the mobile space. So we're running on a lot of operating systems. And most of the integrations are using Windows. Occasionally, we see people using Linux. But the professional integrators are usually pretty flexible about doing what we ask them. They want to succeed. They want to have a system.
42:55They get certified. They don't want to spend a lot of time screwing around. So if they're using Windows and we say, well, upgrade to Windows 11, they'll do it. If they're using Linux and we say, use a different distro, they'll probably do it. We can recommend hardware. So in terms of the overall machine, usually it looks a lot like an ideal machine in terms of the specs are very good because the cost of the hardware isn't that great compared to the whole project. The operating system isn't crazy. It's hopefully not loaded down with a bunch of really crazy mods and add-ons and malware or something.
43:27So it's a pretty good starting environment. We use USB HID for our hardware input. So it's relatively straightforward to wire up just about anything to X-Plane in terms of joystick or control input. We also have a DLL-based plugin system. And we're starting to do REST and WebSocket-based communication with the SIM as well. And we've had UDP input for years. So they have a couple of tools they can get that they can use to get custom data into the SIM. But a lot of the times their joystick hardware is going to just work, whether they manufactured it themselves and got a hid board, or they bought something from another manufacturer.
44:07Usually you can make a config file to further specify how we interact with the joystick and you can tweak a lot of the settings and R settings, but it's pretty flexible. So that usually goes well for them. That makes sense. We've talked about a few different places where you have this interface layer that can be configurable and can let you work across the different types of hardware, graphics, obviously being one input device. Are there other areas where you are ending up handling this diverse set of devices? I know a lot of people, there's a lot of standard libraries out for doing that, but you rolled your own thing.
44:41So what does that look like? We generally try to keep the platform-specific layer as thin as possible because anytime it's platform-specific, we're coding it five times and not one time. So for example, our user interface is rendered into the app in Vulkan and Metal using our own UI kit. And so we're not writing separate Win32 forms and Mac Swift UI code. And then we'd need something else on Linux. Maybe it's GTK or GNOME, or there's a lot of chaos over there. We do something else on Android. We're not doing that. We're willing to say, the X-Plane UI just looks like X-Plane. It's not going to be platform native, but we got to write only once.
45:21And we keep that howl very thin. We don't need that much casing because four of our five operating systems look a lot like POSIX and Windows isn't that different anymore. So the platform story has gotten pretty good. I believe the HID code we're using comes from an open source library, which does give us separate hit paths on multiple platforms because how you talk to the USB driver varies, but that makes it uniform. Vulkan and Metal look different from each other and are abstracted, but Vulkan plays on Windows, Linux, and Android. Metal is on all of Apple's platforms, so we get pretty good leverage from that.
45:57We have a thin set of file system wrappers, but we don't need much. We have a wrapper around the memory mapping primitives to get files in, But again, it can stay pretty thin. And we're not doing tons and tons of file operations, so our needs are relatively straightforward. Racknet abstracts the network stack, and we also have a pretty thin socket wrapper. But Windows looks enough like sockets. So overall, I would say the platform layer, we can keep it thin, and it's been okay. We need a way to get a native window and get some mouse input off of it. But once you write that once, the amount of change in that space is pretty limited.
46:33You know, the Windows code really pretty much runs the same from, you know, when I started, it was still Win32. I mean, people slag on Microsoft, but they have kept a platform running since forever. And I think people don't realize like how much they're holding up by doing that. You know, we feel it a little bit because we've been on OSX where they've changed the ABI maybe like four or five times. They've changed the UI kit several times. They've changed the language multiple times. And you have to rewrite your stack. and your users aren't going, oh, I'm really excited to get the exact same features but rewritten differently.
47:07Like you don't get anything for that. So, you know, as much as we sometimes say mean things about Redmond, like I do recognize the value of having the thing keep working so we can focus on features and not just rewrites. Yes, no, they definitely understand their business consumer, I think, better than just about anyone else in the space. You mentioned features. What's on the roadmap? What types of features are you all working towards? Hopefully, by the time people hear this, but as we are recording, our newest free patch 12.2 should be coming out. That's a patch that's very graphics intensive.
47:45We try to have the patches be in sort of a theme area. Marco, our product manager, kind of looks at things and tries to gather up the features and find things that will be good for users, but also hit a theme and limit the testing. right if we touch everything we have to test everything so the most recent one is a graphics one and there's there's changes pretty much everywhere i don't think i could just rattle all of them off because there's been so much but this is the end of a pretty long probably even a year-long effort to refine the rendering and lighting stack of the sim and really do write what we kind of prototyped in X-Plane 12.0.
48:30X-Plane 12.0 was a difficult release for us because Microsoft had come into the market with Flight Sim 2020 while we were finishing the Vulcan port. And we were completely taken by surprise by this. They really stayed in stealth mode right up till their EA announcement of the new product. And it was just, how soon will you guys have the new version and how many features can you have in it? And that's, you know, that's a pressure cooker for release. So we did all new photometric lighting and clouds and quality was something that kind of got thrown on the floor to get that out there and just show that we were in the marketplace and, you know, people use it and they could see the potential, but there was also a lot of weird stuff.
49:11And so now the team has had the time to go in and really do a great job of working all the bugs out of the lighting and going back to the real world references and using new algorithms. systems and it looks like a whole new sim it's really a big upgrade you know people who have used it in the company you go back to the shipping branch you're like what happened what all the you know now it just looks flat and it's funny how in graphics everything looks fine until you see the new thing and then we hedonically adapt so quick and you go back and go wow we thought that was good and and i i know there was a time when we first saw x-play in 11 and went wow that looks so much better than we could have imagined but then once you've seen x-play in 12 you go back to 11 and go, yeah, that doesn't look right at all.
49:52Like, look at all the things we didn't do right. So the lighting, the color saturation, the clouds have gotten a big update. The interaction between them has been reworked. It's going to be a really nice visual update. Behind that, what I think could be 12.3 for us, but your mileage may vary. We have a number of features for systems and weather that are nearing completion. So better modeling of how clouds work, better interaction with real world data when we can pull it in. I believe we will finally ship the new weather radar, which is a very accurate simulation of the radars that are on aircraft so that the pilot can see storms ahead that runs on the GPU and probes that weather data, but then simulates the returns and the occlusion.
50:38So you get a very accurate view of when there's a storm cloud ahead of you and it's big enough, the pilot doesn't really know the weather behind it because the radar doesn't penetrate it, right? And we simulate that. Or if there's a mountain in a way, you actually get a radar return off the mountain. You can't see the storm on the other side, but you do know there's a mountain. So Philip has done a fantastic job with that realism. We're going to ship that. There's new features for the FMC. And I think we're going to finally ship the new network synchronization layer, at least one part of it. So Pavel has reworked how we synchronize state over the network because you can do land flight, you can do X-Plane plus external visuals.
51:19And for years, we only synchronized the aircraft, which is the most important thing. But as X-Plane has kind of grown a zoo of other stuff, that started to be inadequate. And we get bug reports of, hey, I've got a three monitor, three machine setup, and the jetways aren't in the same positions across the monitors. They don't line up. And it's like, well, that jetway is a simulated object that can extend and go to the airplane and move. And the machines didn't know what the state was on the master machine. So we have new synchronization. So it's going to be exciting to ship that for the first time.
51:53This is one step in a roadmap for improving external visuals. And the team keeps working on features. So we have releases beyond those, which I think Marco will slip my throat if I talk about, but we can see what's coming. The code isn't ready yet. And, you know, it's in progress, but we can say this is going to be a valuable feature for our user. We've, you know, we put shovels in the ground, we're engineering it, and we'll see in what order they come out. Because it can be hard to predict how fast things will get coded because in software engineering, sometimes you don't know what the hard problem was until you crash into it.
52:27You know, if you already fully understood the problem, like it's probably because you already coded it, in which case, why are you coding it again? Like there's just not a lot of repeat problems. when people build a house, it's not that different than the last house they built. And so the estimates can be very accurate because you can use all that experience. And maybe if it's the first house with 3D printed concrete, that's going to be really hard to predict. But if it's the 6 millionth 2x4 framed house, it's a well-solved problem. And in the simulator, we're pretty much never solving the problem we already solved because we already have the code to do that.
53:01So it's always new. So it's kind of always a learning experience, every new feature. Awesome. So we're getting close to the end of our time, and this has been absolutely fascinating. Is there anything we haven't talked about yet that you think we should touch on? I think we've covered a lot of the landscape. I guess the other area where we've seen a lot of change besides Moore's Law drives us to use new hardware, new technology to try to give more to our users. And I should say, I am very thankful to have that problem because I started in video. and when I was at Avid Technology in 1998, video was rapidly becoming a solved problem and it was hard for the company to come up with new ideas that they could sell to their customers.
53:47And what happens is somebody comes into the market with a much cheaper product using the fact that technology is faster and they've got a lower cost engineering structure and you get eaten alive from below, right? The bottom will come up from below and that's where you got to look. and this was after my time there, but I think to a certain extent, their bacon got saved by HD. This need to like go operate all this equipment. It's like, okay, now there's something we can sell to our customers that they really, really need. But other than that, once you solve the problem, you're kind of in trouble.
54:17So, you know, we used to pay for programs to unzip files. Like that was the thing you paid for. And now like we take it for granted. It's built in, right? That's a market that's gone. And that's great for all of us. That's great for users. But as a programmer, you kind of have to look at what you're doing and say, what's driving this? Like, will people want more? So we've been very lucky that our users want higher fidelity simulation and we can go use this hardware and do a better job. The other area where we've changed is kind of new programming techniques and language innovation. So besides moving to Vulkan to use graphics hardware better, we've really changed how we approach concurrency.
54:57And that's also driven by hardware because when I started, dual core machines were exotic. Like there wasn't really a point of coding for them because no one had them. Then we reached a point where, okay, people have dual core machines, separating the loader from the simulator would be worth it. You know, we could load in the background and not pause your flight. Now, 16 cores, sure, 24 cores, 32 cores. People have massively concurrent machines. And fortunately, the language technology is growing too. So we have access to coroutines now in C++. Structured concurrency has become something that people understand.
55:32So you can program with language support for these multi-core machines in a way that isn't the more equivalent of go-to for a concurrent programming. So as we develop new systems, we're also kind of coding them differently. And we're seeing people use structured concurrency in the sim and be able to write concurrent code from day one that's pretty safe, hopefully. I mean, multi-threaded code is always really hard, but this is at least not quite as dangerous. And it's been interesting to see how our techniques can change based on not just better hardware, but better tools and kind of better community learning.
56:07And that kind of helps us to address that challenge. Awesome. Let's call that a wrap.
From the publisher
X-Plane is a popular flight simulator developed by Laminar Research. It features a first-principles physics engine, realistic aircraft systems, and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight simulator so we invited Ben Supnik. Ben is a software engineer at Laminar and he’s been working on
The post The X-Plane Flight Simulator with Ben Supnik appeared first on Software Engineering Daily.