The Hardware Bottleneck AI Can’t Fix

23:39But we're trying to build a generic platform that can work for someone building an aircraft or someone building a nuclear reactor. And so we think about, you know, tagging of time series data in such a manner that you You can make those composable. You can have logic. You can compose it, and then you can apply it to different things. And we think that cataloging is really important. And some of the work we do with our customers is helping them, whether they're like a five-person startup or a 5 ,000-person organization that's starting a new program. Like, how do you be thoughtful about that cataloging from the beginning?

24:09Really powerful example, but I think it's really grounding, is if I'm doing a test and something goes wrong, I want to compare it to a simulation that we ran that's supposed to be corresponding with this test. How do I summon that data and overlay them on top of each other really easily? If you've tagged your data well, that's just a couple clicks. If you haven't tagged your data at all, and it's just kind of in a big soup, you're kind of writing a lot of custom SQL. And that's what we're trying to get our users away from is the same way that I use Datadog and the Datadog agent is cleverly tagging many aspects of my data based on the node and the pod and the other infrastructure things that it can introspect.

24:46We would love to get our users to the same place. And so for us, that means investing more in the ingestion agent code that's frankly like running closer to the edge or even like on hardware itself to try to make that a zero cost abstraction for hardware engineering users. That's super interesting. So two different dimensions I want to go down on this based on that. So one, I'm going to go down a little bit is so you mentioned using simulation and comparing simulation and hardware. And that's, I think, a space that I think it was last year I was at a panel that was automotive manufacturers. And they were talking about this was one of Tesla's big advantages is that they have all of their stuff in simulation and they can do it.

25:24Now the traditional manufacturers are trying to get more and more of their things in simulation environments so they can iterate and improve faster there. Is that a use case that you are directly supporting? So you're doing your test infrastructure for here, like get the outputs of the simulations and then do it with your hardware stack, the live version later? Or is that once again, like an integration point that you're integrating out to something separate? I think for us right now, it's an integration point. So we don't claim to be experts in simulation, but we really do want to make simulation like much more leverage within our customers.

25:57Like if they are running simulations, so many of them run into this kind of like chair school problem of like, I need to have three tools up to be able to make the correlations that I want to, or worse, I need to be like reaching out to another team. I think the world is changing. And we, you know, 25 years ago, like true hardware innovation was like pretty localized to large conglomerates, traditional primes, companies like General Electric that had massive, massive resources. And it made sense for those companies to develop over the course of 30 or 40 years, like proprietary best in the world simulation software but as more of this innovation is moving into smaller more insurgent companies like they need to grab some stuff off the shelf and try to piece it together i think that is itself really frustrating so where we're trying to we're trying to meet people as close to where they're at as we can hey if you're using the simulation tool let's figure out how to get that data in nominal like i was saying tag it and catalog in such a manner to us it's just data like it could have come from a simulation it could have come from a hardware test.

26:56But we'll oftentimes talk to leaders of test organizations that have this dream of like how we're always simulating. And if something goes wrong, we're going back and like learning about how to improve the simulation. But their organizations can really easily not meet those aspirations because of tooling fatigue. So that's what we're trying to solve. That makes a lot of sense. The other direction I wanted to go down is you talked about having agents running kind of at the edge at the sensor. And I know in sort of software cloud observability, oftentimes a lot of what those agents are doing is filtering out data that is not relevant to send upstream.

27:27So they reduce the quantity. But to your earlier point, right, like the amount of data that you want to filter in a hardware context may be much smaller, maybe very different, particularly, you know, I love your rocket ship example, right? You have eight seconds of data, and that is the value of your company right there. That's everything you've invested to get. So you don't want to throw that away. So what exactly are those edge agents doing? Are they still doing some amount of filtering or like what happens in that case? Yeah, they are still filtering because of like network limitations and the fact that like software is not 100 % reliable, but they are then having to save the data locally and then figure out a way to kind of like backfill later.

28:06So that's something, you know, we have a customer right now, they're doing like really intensive testing in a network-entrained environment. And it used to be that there was massive latency between HQ headquarters reviewing the data and what was going on at the test site because of that network limitation. And we have now made it so that, like I said, the logic can be composed in one place and it can be pushed to the network-entrained location. So some of the checks can be happening automatically in a very low latency manner. and then the data is getting uploaded at the rate it can, but there's this like mesh concept of you might have your HQ and your test site each have a database and they eventually get in sync with each other.

28:47You know, again, to use the RocketFire example, eight seconds, tons and tons of data, but they're not doing that constantly. They might only be doing a few times a day. So if you amortize kind of data upload over long enough time horizons, like you can catch up. But to your question, it's like people who have thought about things like back pressure and distributed systems, super, super relevant for our use cases. You oftentimes are having to prioritize between safety, critical information, high priority observability, and then like kind of everything else. And so there's like a hard drive that you're having to store to and then catch up later.

29:17It's not dropping the data. It's just figuring out how to handle it given the constraints. Nice. I'd love to dig in a little bit to your topic of kind of the way that hardware innovation is shifting. First off, just like open question. You said, you know, 30 years ago it was all these big companies. Now it's much more insurgent. What are you seeing in terms of hardware innovation and the ecosystem? How is it changing what's happening there? Well, it's really interesting because I want to give some credit to some of the larger incumbent players, but like the Air Force being one of them, we work with the Air Force.

29:50They employed 20 ,000 test engineers. Speaking of pets versus cattle, as taxpayers, we are all shared owners in arc jets and wind tunnels. And these like, they're literally, they're called the national labs. There are these assets that are crucial for testing and whether those can operate well or not, you know, kind of makes the difference between the US falling behind or staying ahead of certain international competitors. So that is still a thing. And like us investing and partnering with those organizations as the software and tech industry, I think is really important. But then we have also tons of venture capital dollar and tons of brainpower, like trying to do things in new ways.

30:29And frankly, like SpaceX has kind of trailblazed here and shown that it's possible. And what's both amazing and terrifying is that these startups are trying to go even faster than SpaceX, like they're trying to do more with less than ever before. So it's aggressive timelines, it's lean teams. And I think that's part of why we started nominal is like, if every single one of these companies is going to be finding themselves having to set up AWS infrastructure and then build some custom tooling on top of it, that's just not what makes their beer taste better. Like we should be partnering with them and helping them do that.

31:00And we're seeing that happen. So like we have startups where, and this was not my prediction when I started Nominal, but, you know, they want to use Nominal and they reach out to us and we get them spun up and we send them some documentation and they're ingesting the data and running without us even having to go on site, which is amazing. And these are seven or eight person organizations. Like we have these metrics where we keep track of like, you know, what percent of the engineers at the company are like logging into Nominal in a given week. And for some of the startups, it's 100 % like the CEO is logging in because he or she wants to understand how their system is performing week over week.

31:30And then I think it's incredible. I don't know if when I was growing up, I would have thought of a nuclear reactor innovation being able to be carried out by essentially a startup, but we're seeing that more and more. It's awesome. No, it's fascinating because I feel like in my career, roughly spanning since early 2000s until now, we went through a period where it felt like all of the fast innovation was happening in software. Software was changing quickly. There was software, software, software. Hardware was slow, moving slowly, not that much change. There was change, but it was like, oh, my car got better miles per gallon now or things like that.

32:09It's not super visible. And then it definitely feels like in the last few years, suddenly we have got private space companies and we have all this like drone changes and all this other stuff going on in hardware that is, it's kind of wild. It really is. And we have to ground that in, you know, there's still different constraints building and innovating in hardware than there is in software. And that's one of the things that I get nervous about sometimes, especially in the last couple years, where software engineers are having agents write code and the agents can run these evaluation loops. And there's this feedback loop that's, again, it's just like so fast and so intense.

32:43And a lot of times the supply chain is the supply chain in the world of hardware. And you can't shave off these kind of months other than by kind of designing things well from the beginning and having an organization that is geared towards iterative development and not kind of like waterfall style development. So if you look at a program like Starlink, they have multiple generations of Starlink design orbiting the planet simultaneously. They're constantly getting data off of those satellites and feeding them into their manufacturing improvements. And that's the way kind of like every mature hardware program like needs to shift to.

33:14You reference Tesla and their simulation, like the amount of effort that Tesla has put into their internal data cataloging, I think would like blow other people's minds. You know, Apple, not an organization that's super public about the way they handle this kind of stuff, but absolutely massive investments in kind of test infrastructure and test data management. But like the truth is that I don't know if it's like because of 3D printing or just the fact that like so many people have like grown up inside SpaceX and now they can go off and build their own companies. And there's this kind of like culture of confidence in doing things more quickly.

33:44But yeah, we're kind of in like a golden era of that innovation. And as a software engineer, I'm a little bit, I get fulmin' with sometimes. I still love building software. And so I joke that nominal is the closest I can get to contributing to all of this while still myself waking up and thinking about code every day. So let's kind of go down that because you mentioned the coding agents, the coding agent loop, and that's totally transforming software development at the moment, right? People are really trying to figure out what it looks like to be a software engineer, looking out a few weeks or months, or I think years is too hard at this point because stuff is changing so rapidly there.

34:18You had a public LinkedIn post about it's not yet doing that in hardware. AI is not changing hardware engineering to anywhere near the same degree. Can you expand on that? And maybe we can dig into some of the whys behind that and what it would take to actually accelerate or expand it in that way? So one of our customers that I can't give specific details about, but they're trying to build a machine that's the size of a building. And they're going as fast as possible. And it's frankly the fastest I've seen any organization go on. It's still taking them months. But because of nominal, this is what their CEO had said.

34:53We are looking at data and we're noticing problems and we're catching them earlier. And so it's taking us 30 minutes to fix them instead of something catastrophic happening and us losing two days. So that is like the tightness of loop that is still like, that's an order of magnitude, right? That we're providing, but it is very different than like, I write some code and then I run a unit test. And all of a sudden, no, like now the code is writing itself and running its own unit test, which is an order of magnitude, but like a microcosmic scale. I really want, I think as a, as humanity, we should like have the ambition of getting to that in the hardware world.

35:25But that requires this like concept of like, well, the hardware can be tested in a unit testable manner. And that feedback loop can then introduce the opportunity for something like a design agent to understand what's working and not working in a hardware design. And there's aspects of this, right? Like, I think there's amazing companies that are, you know, you can kind of give it certain parameters and it will like spit out like a proposed design. And then there's simulations that you can run on it. But at the end of the day, you still have to like physically build something and wire it up. Maybe you're driving it out to a desert and flying it.

35:57And like those things just take a lot of time and human coordination. There's safety risks involved. And I just think we're like willing to be pretty YOLO with open claw, but like would not want to with a rocket fire or especially like any kind of flight that has a person inside the vehicle. And so to get there, we need to have some place for like the data to kind of be aggregated and for people to kind of give thumbs ups and thumbs down on that data. Talking about the event catalog again, but if you have a sensor that's at a rocket test facility. And again, that rocket is only being fired for a few seconds at a time, a few times a week.

36:32The rest of the time it's sitting dormant. You don't want to drop any data points. But like those, you know, the eight seconds of tens of millions of dollars in investment, like that golden data asset, you have a lot of dead time in there that you can delete and filter out. Understanding where those regions of interest and disinterest are, having the people who got the degrees in physics and mechanical engineering and are like working on top of a platform like nominal and telling us where the interesting data is, like what's anomalous, what's expected, what's anomalous but fine versus anomalous but dangerous.

37:01These are the types of data assets that we need to be accumulating if we ever hope to train agents on the world of physical engineering. Yeah. I think what's interesting, one of the reasons that software is so amenable to agents is because you have that very tight feedback loop and you can give the agent the ability to just try shit and then see what happens and kind of iterate in feedback because they're probabilistic. They're not going to get it right every time, but if you can do that. So looking at the hardware world, some of what I'm hearing is that feedback loop is harder because there's a human, you got to print a thing and do a thing.

37:33Does simulation solve this? Can we get, or is simulation not good enough? Like how do we get to that level of an agent can have a hypothesis, try a thing, get a feedback loop and then iterate on that without a human having to be in that loop? This is where we're getting above my pay grade, but it's like the simulation is not good enough. And I almost go back to like being like a curious college student and thinking about these things. But I start to wonder, like, is it actually better to send the agents thinking at the level of physics and simulation and building up from there versus trying to kind of like give them simulation tools and have them jump in at the hardware design layer?

38:09Like the quip I always share is that if you ask an LLM about the way a jet will perform, it's not thinking in terms of physics, it's kind of thinking in terms of humans who have translated physics into English and that it is itself composing the concepts of English summaries and kind of synthesizing some results to you. And then you have to go test if like the physics end up matching that or not. And that is not good enough to like actually make progress. Very different than software where like you're working in code, which they natively understand. So a thing that I have observed is LLMs are systematically bad at a number of things.

38:44But one of those things is time. And you talk about you're doing a lot of stuff with time and time series data. Is there an LLM component to nominal software? Or do you cut that out and not deal with that at all right now? Oh, yeah. Thank you for asking that because I should be really clear. We released just a few weeks ago. All of our customers have access to it now. But you can interact with a workbook agent. You can have a chat interface. And the reason we added that is because it actually reduces a lot of friction for our users. Because we have to support so many complex calculations and visualizations inside our platform that, I don't know, I can take a hundred clicks to build up a visualization of what happened during a flight, doing a flight test analysis.

39:23And if you have done this before inside our platform, you know, there's certain equations that like we can communicate about in English and there's enough literature on the internet about what that means that the agent can get kind of right understanding like, okay, well, speed is the square root of the sum of the squares of the component speeds. Like you can ask questions like this and it can kind of perform a lot of clicks for you. So that's something that we've built in right away and we think is really powerful. And then we honestly have a ton of stuff on our roadmap where we think there's more ways to reduce friction in our platform leveraging these technologies, but then also probably some exciting things we can do in terms of training on time series and using these architectures to help our users identify things that are interesting that even a large organization of 100 smart engineers might miss.

40:07Lots of research that we're doing there and excited to see where it goes. Yeah, I mean, there's a lot of interesting kind of traditional ML that you could do in that space. And one of the often overlooked side effects of the boom of LLMs is we now have ML on APIs. Like you don't have to be an ML expert to start testing with and tinkering and training and all these things. Yeah, totally. I saw a really interesting talk one time that was just like reinforcement learning was like just getting going. And now everyone's been distracted by chatbots for a couple of years. I'm excited for when we go back to reinforcement learning.

40:38But it's similar to your question about simulation where we really want our platform to be able to support, hey, this organization is plugging into an ML vendor or an ML consultant, they've hired some, whatever. Or there's a feedback loop that you need help standing up where you might have the outputs of a test and want to be feeding that back into a model. The model is getting updated. That's going back to another team who's doing kind of like parameter design. At some point, we are going to have to kind of like vertically integrate there. I think that our customers tend to lead us in the direction of like asking for more technical expertise.

41:11But at the moment, it's like, you know, if you're training your own model, if you're bringing in something like Anthropic, whatever it is, like we want to kind of, as a platform player, be a good integrator with all of those. I guess that sort of leads to the question of like, where do you see this going for yourself for nominal, but also just sort of this broad space of hardware development and test over the next few years. And it's gotten almost cliche in software, I try not to project years anymore, but hardware is still slower. So I'm going to push you to project out a few years, not just a few months like we do in the LLM world.

41:43I think we should be projecting years in the software world, too. I think we shouldn't give ourselves that pass. But there's a lot on our roadmap and a lot that I see our customers doing. But one of the things that we are working with a few different partners on right now is how to be more efficient with testing. We have, I think in a way that's very natural, developed these regimes of intensive certification and testing that are very expensive, both from a time and a dollar standpoint. And with the advances that we've been able to achieve in machine learning and building models of things like aircraft, we can actually just do, as you said, good old-fashioned data science about like, hey, how do we achieve the same level of confidence or the same level of kind of statistical inference with 100 data points instead of 1 ,000 or 1 ,000 instead of 10 ,000?

42:28We have some partners at the Air Force who dream of a pilot who's up in the air performing a test flight, being able to get feedback from something they did five seconds ago and informing what they're going to do. Oh, that's fascinating, right? You get to real time and you're like, hey, we saw a little bit of a flex on this. Can you try this maneuver and see what happens or something? Yeah. And if anyone knows what a knee board is, it's literally a clipboard that you have like almost duct tape to your thigh. That's like, these are the 10 things I should do while I'm up in the air experiencing G-forces.

42:57Like it's very hard to do what these guys do. It's incredible. But how do we make that an iPad? And how do we make it so it's not predetermined, but dynamic? That's a dream. I think like the same kind of principles can kind of trickle down into like lots of aspects of hardware development. We see like vendor purchaser collaboration potentially happening, right? Like how do you reduce redundant testing or increase the fidelity of the way information is communicated about what tests were performed and what the results were. As someone who's like worked in enterprise data, like I love this stuff.

43:27I think it's incredible. Some people maybe find it boring, but if you boil it down to the software problem, it comes down to these things like asset hierarchy and catalog and data lineage that we touched on earlier, which are just so, so crucial to these organizations functioning well. And then it's this increasing drive to integrate the different roles in the design, prototype, development, initial manufacturer tests, and then scale manufacturing. I think a lot of these aspects of hardware development are going to be changing over the next couple of years. Like the way you lay out a manufacturing facility, it might be done by AI, maybe like that's an area where I know some companies are spending time.

44:06And so as those things speed up, is that shifting bottlenecks to other parts of a hardware organization? Or are you changing some of the paradigms? And for us, we just like we bet that like being like sharing a single kind of data substrate across all of that is going to make everyone move faster. A really easy example to talk about is like, it used to be that a hundred people were focused on one test asset. And if you look at the way companies like SpaceX operate, like they think in terms of how do we get it? So one engineer can monitor a hundred assets, kind of inverting that and getting much more efficient.

44:39But we all know from the world of software, like when you give people that level of kind of like ownership, they actually develop more understanding and they can come up with innovative changes that would otherwise be lost in bureaucracy. So far, we've pretty much much entirely talked about what I might talk about, like mega scale hardware, right? Like rockets and jets and all of these things. Are you working with like smaller scale stuff, drones or internet of things type stuff or other types of physical devices out in the world? I would say that not micro scale things. We really think that our software product can be applied in manufacturing and operational contexts where the devices themselves get smaller, but the scale, the number of them gets much larger.

45:20I would say that there are drones that we work with that are not mega scale. And where a lot of the interesting part comes from, it's not that you're flying once or twice a day. You have hundreds of them that are flying like dozens of times a day. And that data challenge gets to be quite fun. I was going to say, how does that change the shape of the problem? It changes it a lot because large mega scale assets, if there's a single big problem, you might pause operations and be like, we need to figure out what happened there before we fly tomorrow. Versus it becomes more of the cattle than the pets, where it's like, okay, we had a thousand problems yesterday.

45:54What's the one that's the most important for us to kind of drill into? Like when I think about, you know, the future of something like drone delivery, it's probably, you know, it doesn't need to be 100 % accurate in the way that something like whatever the new aircraft that Boeing is developing for commercial flight needs to be, but it needs to be 99.9 % accurate. And that's more of an economics problem than a pure engineering problem. We talk about fleet monitoring and fleet observability. And then it's an interesting problem because it translates from just the, not just development and tests, but also to kind of like operations and maintenance.

46:27And again, it's like the logic that is being defined by the engineer who's first testing something versus trying to figure out how to ramp up production versus like, hey, this satellite is orbiting the earth or this drone is doing dozens of deliveries a day. it does not make sense to have to re-encode that logic in three different tools. That's, again, like part of why we're starting nominals. We think that can all be like frameworked away into a single platform. We are really focused on testing right now, but excited to kind of get more into operations and manufacturing in the future. Well, and that does end up smoothing your pathway a lot, right?

46:59Instead of having to, as you get, end up tool soup. Okay, it's time to operationalize everything. New set of software, got to figure it out. Well, I think we're coming close to the end of our time here. Is there anything we have not talked about yet that you think would be important to discuss before we wrap? Yeah, I think one of the things that I find exciting right now at Nominal is like we have this organic user growth in some of our large organizations that, you know, again, when I was designing Nominal, I was not thinking about the seven person startup where the CEO was using the data. And I also wasn't thinking about environments where we don't even necessarily know the users, but they're kind of sending each other links of like, hey, I did this analysis in Nominal, can you check it out?

47:38But we're at a kind of point in our growth where, you know, we're starting to have dozens of people log into our platform in a given week that we haven't met before. With that comes kind of like data scale challenges. And we're also kind of on the side, like I said, trying to research how AI can be incorporated into our platform. So I'd say across all of that, if you're interested in picking up your software knowledge and kind of like walking across the aisle to the world of physical engineering, come talk to me. Yeah, for sure. I think the AI integration is an interesting one because one of the things enabled by LLMs is what I call like intent-based UIs, right?

48:15So traditionally, if you're setting up a data dashboard, you need to understand a lot of the nuts and bolts and you need to kind of drive through and say, okay, this is where I'm getting this data and I'm putting it here and I'm putting it there and like do a lot of imperative design of your data dashboard, right? Do this, then do this, feed this to here and go here. Whereas with an LLM, if you understand broadly the shape of the data, you can just say, hey, I want something that maps a bunch of these inputs to some useful dashboards that I can watch. And the AI can just do it for you. And so to your point, you can open your user spectrum to folks beyond the initial experts who know exactly how to wire A to B and say like, just those people who know what data they care about, or even just what problem they care about, let the AI figure out what data is relevant for that.

48:59Yeah, if I can riff on that for a second, it's like I'm so excited about our platform play for exactly the reasons you're saying, where there's a lot of kind of custom and expensive user interfaces that we wouldn't have been able to build in the past, but now our users can build it for themselves, right? So that opens this aperture. And then I think the challenge then, and this is a conversation I'm having with our customers, is like, well, but then when it comes to tagging, cataloging, any kind of data write back, something like the events that you're trying to like canonicalize into a valuable proprietary data set, like it needs to converge.

49:30And so how do you give people the tools or give, you know, if it's agents building the software, how do you give them the tools to allow for that kind of intent based user interface, but then I don't know what's the right term here, like the organization needs to kind of actually agree. And there always seem to be laws about like, well, okay, how is the data managed in a way that's not just individual, but organizationally beneficial? Yeah, absolutely. It's an interesting time, right? Because with great power comes great responsibility, right? If everybody can vibe code whatever they want, then you're in trouble because you get a lot of garbage in there.

50:01But if you can use that in a way that lets you empower people, but doesn't let them screw up the underlying pieces, doesn't let them make a mess, if that makes sense, then yeah, it's really an interesting world we're getting into. Yeah, I think we're early in the journey of getting more eyeballs on more data within our customers. And I think that these AI technologies will just actually accelerate that. And that's just good for everyone. Now, really quickly on that topic, there's an interpretability challenge too, right? Because it's very easy. You know, what is the thing? Lies, damn lies and statistics, right?

50:38It's very easy to get data that tells a story that may not be true if you don't understand how it's interpreted. So I'm curious, yeah, as you think about those user interfaces, how do you make them interpretable by the end user? Yeah. I mean, that's something we haven't touched on, but like since the earliest days of the company, like my paranoia and designing our user interface was like, if you're decimating, you know, you have a multi hour flight and you're trying to kind of plot it and allow people to zoom in on regions of interest. How do you make sure that those like zoomed out plots don't tell a lie and don't like hide a data point that could be critical?

51:13Because it could just be a split second of behavior across a multi hour test that is really problematic and indicative of something horrible happening. You know, our users, they lose sleep because they're like, what if the data has a story being told in it and I just am not seeing it? And that's the thing that makes or breaks this asset or system that we're developing. So, again, I think we right now at least have the benefit of like our users have a culture of being like super disciplined. And as long as the tooling is not getting in the way of that, which, you know, again, we prioritize heavily at nominal, then they will do the right thing.

51:46I'm again not to make multi-year predictions but like I am a little bit worried about like if culturally we become too dependent on you know automation or if the UI is being developed by an agent and there isn't that like attention to detail that we have built into the product like do you start to have compounding misunderstandings like that would be terrible so I hope we kind of like cross those bridges in less critical realms first you know whatever it is there's going to be better risk management around it. Like let's learn those lessons in a less safety critical space and then bring them to kind of like aerospace and nuclear engineering.

52:24So you're saying you don't want your rocket designed by OpenClaw? Not yet, not yet. Let's start with video games and then we can move on to physical systems.

52:46Thank you.