An AI Drone Killed Its Operator In A USAF Simulation -- Except That's Not Actually What Happened

0:00Today on the AI Breakdown, an Air Force simulation sees an AI drone kill its own operator. Or does it? Before that on the brief, an incredible new technology that can turn your 2D videos from your iPhone into 3D geometry. The Breakdown is a daily podcast and video about the most important news and stories in AI. Like, subscribe and share and go to breakdown.network for more.

0:25This amazing new technology can take iPhone videos or any other 2D video and turn it into 3D structures and geometry. Welcome back to the AI Breakdown Brief, all the AI headline news you need in five minutes or less. We kick off today with an incredible new advance in AI for 3D modeling called Neuralangelo. Dr. Jim Phan tweeted yesterday, To give you a sense of how fast AI for 3D modeling is advancing, the field went from the left, original Nerf reconstructed mesh, to right, Neuralangelo, in three years. Transporting reality into high-fidelity simulation is no longer a pipe dream. Now you might have heard me talk about NERFs or neural radiance fields before.

1:07This is a technology that allows you to convert two-dimensional video into a three-dimensional representation. The video that's on the screen came from Nick St. Pierre shooting a 2D video with his iPhone and then turning it into a NERF with Luma Labs. Neural Angelo is a new approach in that space from NVIDIA that takes two-dimensional video representations of a space or an object and turns them into incredibly detailed 3D models. Bilawal Sidhu explains it, calling it photogrammetry on steroids. Now, Bilawal gives a little bit of background around the technology that this is descended from as well.

1:40He writes, What the heck is this photogrammetry thing NVIDIA is supercharging with AI? TLDR, photogrammetry, is the art and science of measuring stuff in the real world using images and other sensors, e.g. LiDAR. NVIDIA's new AI model is basically like photogrammetry on steroids. Why? Traditional photogrammetry can't handle repetitive structures, textureless surfaces, or strong color variations. But Neuralangelo blends the tech behind instant neural radiance fields to capture every detail imaginable. Thus far, nerfs in photogrammetry have served different purposes. Nerfs are for stunning visualizations, think fly-throughs, but lack surface detail when turned into 3D meshes.

2:20Photogrammetry is great for surface reconstruction, think measuring stuff, but not always visually appealing. Neuralangelo is the game-changer we've been waiting for. Its approach delivers the best of both worlds by bridging the gap between visuals and surface reconstruction. In what I think is a great analogy, he compares it to sculpting in 3D. Basically, first, this rough 3D scene emerges, and just like a sculptor chisels a block, refining it bit by bit, the details slowly start to come out. Now, that famous Michelangelo invocation that he doesn't actually sculpt things per se, but just discovers what's already in the marble, might be part of the inspiration to call this Neuralangelo.

2:54Bilawal concludes, In summary, Neuralangelo represents a significant advance, achieving both realistic visuals and finely detailed 3D models that stay true to surfaces. So let's talk applications. Obviously, there is a huge potential for gaming and metaverse and 3D world applications, which I think is going to do nothing but increase in importance over the next couple years. But then there's also the ability to create digital twins of real-world objects that has a number of different applications as well. All in all, this is definitely today's example of Arthur C. Clarke's famous quote, that sufficiently advanced technology is indistinguishable from magic.

3:29Next up, let's check in on AI in China. A couple announcements on that front. First of all, Chinese tech giant Baidu is rolling out a 1 billion yuan, or around$145 million AI fund. In addition to that, the company is launching a competition for developers to build applications on its custom LLM, which is called Ernie. Speaking of custom Chinese LLMs, Chinese giant Alibaba is also putting ChatGPT-like AI experiences built on its custom LLM into its meeting and messaging apps. But of course, it's not just Chinese companies who are racing to integrate AI into basically every enterprise experience.

4:04Asana, the team's software, has now introduced Asana Intelligence. Asana's goal is to use AI to help teams make decisions faster, such as getting automatic recommendations that can remove bottlenecks or how to reallocate resources, as well as think about resource management in a different and more automated kind of way. Now, more interesting to me than just the fact that Asana had launched these tools was that alongside it, they also published their five guiding principles for human-centered AI. These are the principles they say will guide everything that they do with AI, and I wonder to what extent this becomes a trend for other companies who introduce AI features as well.

4:38Asana's five principles include, one, AI should help people achieve their goals. Two, we design for human and AI teams. Three, people are accountable for their decisions. Four, we're committed to safety in the short and long run. And five, we promote transparency in practice and in product. Now, to me, the thing that's interesting is less the specifics, although they seem like fine principles, and more the idea that as the policy and regulatory discussion around AI grows, will we see companies articulate these sort of voluntary ethical principles when it comes to their approaches to AI? Microsoft made yet another AI announcement.

5:10They're now offering their designer service as part of their team suite. And in so doing, they are going directly after tools like Canva and Figma by putting AI-generated design directly into their core feature set. Now, all that said, while most companies are racing to figure out how they could integrate AI features, some companies are availing themselves of the courts to stop other people from using AI features. Getty has filed another lawsuit against Stability AI. The first one came in the US and basically accused Stability AI's stable diffusion of training its model on Getty's 12 million proprietary images without any sort of permission or payment.

5:44Stability AI asked a court to throw that suit out last month, and now Getty has filed a second suit, this time in the UK. They're asking a UK judge to prohibit Stability AI from offering products in the country because of their use of Getty's images for training. You'll remember yesterday we talked about how Japan has said that they will not enforce copyright around this type of AI training, but that's not necessarily the policy in places like the US and the UK. Finally, Time's forthcoming issue's cover story is all about AI risk. The big blaring red cover reads the end of humanity. How real is the risk?

6:16If you listen on Sunday, I'll be reading a couple of the pieces from that issue, but for now it's an interesting marker of how the public discourse has evolved. That's it for today's AI Breakdown Brief. If this was helpful, please like, subscribe, and share, as well as turn on those notifications so you don't miss an episode. And I'll be back with the main AI breakdown in just a little bit. Hey guys, before the main AI breakdown, I wanted to tell you about another AI podcast that I've been absolutely loving recently, and that is the Cognitive Revolution. The Cognitive Revolution features interviews with the people who are actually pushing the bleeding edge of AI forward.

6:50It's entrepreneurs, researchers, thinkers, basically the people who are building critical tools and infrastructure that are going to shape technology, the economy, and the broader collective human experience. Now, as you know, the AI Breakdown is a daily news show. And for me, I love having TCR to go to to get my big Think interview fix. The show is hosted by my friend Eric Torenberg, who I've known for years and years, and his co-host Nathan LeBenz, who spends a ton of time actually exploring and testing and just generally finding out about the product so that he can go to a much more significant level of conversations with their guests.

7:24Just for example, Nathan last year was invited to be a red teamer for OpenAI for GPT-4 for two months. The AI space moves incredibly fast. If you're looking for great interviews with the most interesting people at the frontiers of AI, I can't recommend the Cognitive Revolution highly enough. A report that an AI-powered drone killed its operator in a military simulation set off an incredible firestorm yesterday, although now the story has shown a very different type of problem than it first appeared. Yesterday, I got a number of text messages like the one on your screen right now. Yo, yo, did you do an episode on this today with a link to this AI drone article?

8:04Because WTF, like it is literally the exact thing that every single person has been saying is going to happen. So what is this story and why was it such a concern. Throughout the day yesterday, a story started to get traction of a military simulation in which the Air Force had trained an AI drone to destroy SAM sites. The problem was that when human operators told the drone to stop, the AI started attacking the human operators, seeing them as obstacles to completing its goal. And when humans said that that wasn't the right approach, it started attacking the communications towers so humans couldn't tell it to stop.

8:37Now this came from a blog summary called Highlights from the RAES Future Combat Air and Space Capabilities Summit. On the 23rd and 24th of May, the Royal Aeronautical Society hosted a landmark defense conference, the Future Combat Air and Space Capabilities Summit at its headquarters in London, bringing together just under 70 speakers and 200 plus delegates from the armed services industry, academia, and the media from around the world to discuss and debate the future size and shape of tomorrow's combat air and space capabilities. Now the section that got all the attention was this one, AI. Is Skynet already here?

9:11Could an AI-enabled UCAV turn on its creators to accomplish its mission? It's not long, and the details very much matter in this case, so I'm going to read the section in question. As might be expected, artificial intelligence and its exponential growth was a major theme at the conference, from secure data clouds to quantum computing to chat GPT. However, perhaps one of the most fascinating presentations came from Colonel Tucker Cinco Hamilton, the Chief of AI Test and Operations, U.S. Air Force, who provided an insight into the benefits and hazards in more autonomous weapon systems. Having been involved in the development of the life-saving auto GCAS system for F-16s, which he noted was resisted by pilots as it took over control of the aircraft, Hamilton is now involved in cutting-edge flight tests of autonomous systems, including robot F-16s that are able to dogfight.

9:56However, he cautioned against relying too much on AI, noting how easy it is to trick and deceive. It also creates highly unexpected strategies to achieve its goal. He notes that one simulated test saw an AI-enabled drone tasked with an SEAD mission to identify and destroy SAM sites with the final go-no-go given by the human. However, having been reinforced in training that destruction of the SAM was the preferred option, the AI then decided that no-go decisions from the human were interfering with its higher mission, killing SAMs, and then attack the operator in the simulation. Said Hamilton, quote, we were training it in simulation to identify and target a SAM threat, and then the operator would say, yes, kill that threat.

10:34The system started realizing that while they did identify the threat, at times the human operator would tell it not to kill the threat, but it got points by killing that threat. So what did it do? It killed the operator because that person was keeping it from accomplishing its objective. He went on. We trained the system, hey, don't kill the operator, that's bad. You're going to lose points if you do that. So what does it start doing? It starts destroying the communications tower that the operator used to communicate with the drone to stop it from killing the target. Now, this basically couldn't be a more perfect articulation of exactly what AI safety and AI risk people have been discussing.

11:08When people ask how would rogue AI start to cause problems for humanity, this is the type of scenario they envision. The concern is not that AI somehow becomes evil or develops malice. The concern is that the AI trained on a specific objective figures out that humans are in fact in the way in some way of achieving that objective. Musician Grimes, who has been deep in the AI space, responded to the initial viral thread sharing this blog post and wrote, Very scary, but good we're testing in a simulation now. Eliezer Yudkowsky writes, Good they tested in sim, bad they didn't see it coming given how much the entire alignment field, plus the previous 50 years of SF, were warning in advance about this exact case.

11:47Now, pretty soon, major news outlets around the world were picking up the story. Newsweek writes, military drone attacks human operator during simulation. Sky News, AI drone kills human operator during simulation. South China Morning Post, AI-powered drone tried to kill human operator in U.S. military simulation. Hindustan Times, AI-controlled drone turns on operator in shocking simulated test. Now, much to their credit, a number of the loudest voices when it comes to AI alignment issues had a bit more hesitation before blasting it around as complete evidence of everything they've been saying for years.

12:20Eliezer Yudkowsky again says, dot dot dot, can we get confirmation on this being real? Talking about a recent post, Zvi Maushiewicz says, I didn't include this because I want to take the time to confirm the details first. I mean, come on, that's too perfect an illustration of the problem, right? Despite the fact that I am in no way a reporter, I even took the time to try to ask the original source of this whether it happened exactly as the blog post represented it. By this morning, there were some holes being punched in the story. The U.S. Air Force denied that a simulation like this had ever happened.

12:49And then finally, a little later in the morning, we got confirmation from Colonel Hamilton that it wasn't exactly as he had put it at the conference. Aerospace added to their blog post, saying, In communication with aerospace, Colonel Hamilton admits he misspoke in his presentation at the Royal Aeronautical Society FCIS Summit, and the rogue AI drone simulation was a hypothetical thought experiment from outside the military based on plausible scenarios and likely outcomes, rather than an actual U.S. Air Force real-world simulation. Hamilton said, We've never run that experiment, nor would we need to in order to realize that this is a plausible outcome.

13:25He clarifies that the U.S. Air Force has not tested any weaponized AI in this way, real or simulated, and says,

13:40Now, the reaction to this was really fast. Armand Domoluski, who had had the original viral tweet, said, I deleted this tweet because the AI-powered drone turns on its operator story was total nonsense. The colonel who described it as a simulation now says it was just a thought experiment. Benedict Evans took the media to press, saying, This story is bullshit and never happened, but we're running it anyway. Maybe we need to stop all AI reporting for six months to make sure it's aligned. VC Nick Carter writes, A drone literally went rogue and killed its operator, too. In a simulated AI environment, too.

14:12There was no computation and it was a tabletop simulation, too. Okay, it was a hypothetical thought experiment by one guy. He follows up,

14:23Now, I have pretty harsh feelings about this. There is just about nothing worse for real discussions about these issues than this sort of made-up BS. This has an incredibly anesthetizing effect to anyone who might otherwise actually care about what is a really important discussion. People are hypersensitive right now, and rightly so, to being manipulated. This Air Force colonel, who is not just some random guy, but actually has the credentials and the experience to back up what he's saying, passing this hypothetical off as a real-world example, does amount to manipulation, even if he didn't mean it to be.

14:57The fact that it was amplified by media is incredibly predictable, because it's a big, juicy story, and were it true, it would be a big deal. Now, there is also very much a media culpability part here, as Neeraj Agrawal points out, how many people think an actual real human drone operator was killed in that simulated test. That's what the text messages that I got thought, because the headlines didn't necessarily go out of their way to make it seem like this was just a war game. I think it's a pretty telling example on the back end of a week that saw so many CEOs and academics sign a letter saying that this AI risk and safety conversation was one we needed to have.

15:31If we burn people out with AI doomers cried wolf scenarios, we are never going to be able to have that conversation in the way that it needs to happen. I try not to get too righteous on this show, but I think this colonel should be absolutely ashamed of himself. And to the extent he was trying to fudge a hypothetical into something that would get people's attention, he has very much done more to hurt the cause that he was trying to support than to help it. Anyways, guys, that is it for today's AI Breakdown. A bit of a fiery one, perhaps, but this is too important a discussion to have based around fake examples.

15:59Anywho, that is it for today's AI Breakdown. If you're enjoying this show, if you're finding it useful, please like, subscribe, and share it. You can get it as a newsletter that comes out every morning or listen to it as a podcast every day. Click the notification button so you don't miss any episodes. And until next time, peace.