AI Safety Language Is Destroying the Debate | Steven Sinofsky

0:00Theo Jaffee:The AI people are making it impossible for anybody to understand what they've done. And they're using words like, well, the AI failed to be aligned. Okay, what does that mean? What it means is there was a bug in the software. When Word ate your file and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.

0:32Steven Sinofsky:When AI does something we didn't expect, is it misaligned? Or does the software just have a bug? A16Z board partner and former Microsoft Windows president Stephen Sinofsky joins Theo Jaffe and Sofia Puccini on MTS to argue that the language around AI safety is making the technology harder to understand. Drawing on decades of building software, Stephen explains why AI labs need better telemetry, debugging, and incident reporting, and what today's industry can learn from earlier software failures, computer viruses, and Y2K. They also discuss why terms like alignment, goal-seeking, and rogue agents can distort the policy debate, and why treating AI more like software could lead to clearer conversations about both safety and regulation.

1:23Steven Sinofsky:We are live with Stephen Sinofsky, who is a board partner at Andreessen Horowitz. He's a seed investor, author of the Substack and the book Hardcore Software. Previously, he was the president of Microsoft's Windows division, engineering and marketing across Windows, Windows Live and Internet Explorer. And he posts on Twitter at SteveSI. Stephen, welcome back. Howdy. so we were just talking about uh we're in dc right now at the fai office and fai yeah we're at fai and we've been talking about uh all of the new regulation and new laws that have been proposed for the ai industry over the last few weeks uh including the stop rogue ai act which was proposed by uh josh gottheimer of new jersey and you're offering some thoughts on this uh what's What's your take?

2:12Theo Jaffee:Well, sure. Well, right now we're in the maximal phase of like, let's just throw tons of legislation out there. So real quick, I'm just going to take you back. I can't believe we're going to take you this far back. This is going to seem like infinity years ago in another dimension of time and space. But in 1983, I was a freshman in college. And one day in the fall, like, and there were two people in our 100-person dorm that had computers, me and another guy. And that was it. and there was nothing to do with them. Nobody knew why we had them, blah, blah, blah. And one day, literally, G-Men in suits crashed our dorm, confiscated the other guy's computer, and arrested him.

2:56Wow. And it turns out,

2:59Theo Jaffee:fast forward a little bit, blah, blah, blah, he was part of a hacker group, one of the very, very, very first hacker groups called the Inner Circle, or the inner ring. And they were hacking this thing called GTE TeleMail. GTE was a big communications company. And it was one of the very first online mail services that you used a modem and you dialed up and you could just mail other people on TeleMail. Who were some of the clients of it? The United States government and NASA in particular. And immediately after this series of arrests, they arrested people all over the country in this coordinated sting operation.

3:39Theo Jaffee:I mean, imagine, I'm a freshman in college, and they just carted this guy off with this IBM PC XT that weighed 50 pounds at a giant monitor. And it was like a freshman in the same, we were the computer science majors together. And it turns out that there was no crime committed. It wasn't illegal to do what they were doing. There was no law that they were violating. Oh, wow. And suddenly, there were all of these hearings. And in the hearings in Congress, they're literally playing clips of the new movie War Games and explaining like what this is happening. Like these kids in high school are going to take over the government and launch all the nuclear weapons.

4:19Theo Jaffee:And that's what really happened. And it took several years before the first federal crime statute for hacking. And Ronald Reagan signed that. And that's what became illegal. And then five years later, four years after that, I'm at my first homecoming and I go upstairs to the computer science building to go see my old advisor like the day before homecoming. And there's a press conference going on. It was the weirdest thing I'd ever seen, a press conference in the middle of Ithaca, New York. And it turns out that was the press conference to announce that they've found a computer worm called the Morris computer worm that had originated from Cordell.

4:58Theo Jaffee:And that was the very first prosecution under this criminal act. And so this stuff for me goes way, way back. So I'm very comfortable with all the congresspeople posing legislation and having hearings, being hysterical. But right now we have this huge problem. And well-intentioned people like Congressman Goheim are getting on TV in the morning and saying, we need a bill to stop rogue AI agents. And the Stop Rogue AI Act. and you listen to all of this and you realize we have a huge problem. And this huge problem is just making my head explode, which is the AI people are making it impossible for anybody to understand what they've done.

5:40Theo Jaffee:And they're using words like, well, the AI failed to be aligned. Okay, what does that mean? What it means is there was a bug in the software. The software did not do what we thought it would do. But when you say misaligned, it puts all these characteristics on AI that it was thinking of doing something, that it had to make a moral judgment, that it had roles and guidelines to do things. And it's like, wow, we had bugs in spreadsheets where they didn't do what we thought that they were supposed to do. And we didn't sit around like in a circle and pray to the gods of alignment that the spreadsheet would do math correctly.

6:24Theo Jaffee:like what you know when when word ate your file and deleted all your content we didn't think that demons had taken over word and made it do things against the will of man and this is a huge huge problem it because it's literally nonsense what they're saying like there was a bug like when you you you what software doesn't do what it's supposed to do it's a bug now there's different kinds of software there's simulations and there's models and there's emulation there are all these ways to decide what to do. And just because it decides to do it, not based on step one, step two, step three, if A or B do step five, but just based on statistics, that doesn't mean you eliminate the idea that it should do what you want it to do.

7:09Theo Jaffee:It means your statistics are wrong. It means that you guess a word that should come next that shouldn't have come next. And that's just a bug in your product. And we've got to get to a place where people are talking about software like it's software. You know, if full self-driving blows through a stop sign because the software interpreted the stop sign as like a Christmas tree, we don't sit around and say, oh, it was not aligned with the idea of, you know, December holiday seasons and road signs. We actually say it's a bug and it's a very dangerous bug because that stop sign means something. And Tesla built in all of these tools and Waymo.

7:54Theo Jaffee:They have all of these, all this telemetry, all these diagnostics, all these extra cameras, all of these things that you don't normally put in a spreadsheet, but you put in software that life depends on it. And it's making mathematical, statistical decisions on things. What I see is the AI models are still in the research project phase. They don't have all the tools and all the telemetry and all of the things that you would normally put in software doing important things. And I lived through this. We went through a whole 20 years of building software where we actually never really knew why it crashed.

8:35Theo Jaffee:It just crashed and we went, oh, software crashes. Let's make a list of all the places it crashes and fix the top 10. And then one day we said, well, why? We literally didn't say this. Someone said, why don't we just write a program such that when it crashes, it uses this new internet thing to tell us that it crashed. And suddenly we had telemetry and suddenly we had more bugs to fix than we knew it ever existed. Like we literally went from like, we're shipping, we have no bugs, we're ready to go to, oh my God, we have enough bugs that if we just stopped work, we would just fix bugs for the rest of time.

9:14Yeah.

9:15Theo Jaffee:So when you talk about an AI pause, of course they should slow down. They should stop adding things and go and add the telemetry, the tools, the logging, the step-by-step debugging, all of the stuff that you would add if your software did anything at all important and you cared if it did it right. And you read all the bug reports from OpenAI that they just put out yesterday and it's super clear that they just, they don't know. And so they, like, if somebody had handed me a bug report And this is, here's a real bug. In 1987, the first version of Excel for Windows had a bug in it that somewhere on the disk, it wrote the word sin dogs.

9:57Theo Jaffee:Just S-I-N-D-O-G-S in a particular Windows settings file. It was like Excel got taken over by demons and it wrote this word to the file. Nobody knew where that word came from. Everybody was looking through all the Excel source code to find the word send dogs and to figure out, and no one could find it. And so somebody said, you know what we're going to do? We're just going to run Excel brutally nonstop doing a million different things for days on end. And so one tester wrote a script that created graph after graph after graph and printed it on an old school dot matrix printer. And it turns out if you did that for like two days, it would write send dogs to that one spot in the file.

10:42Theo Jaffee:Yeah, and I thought it was all contextual, and it was based on how much memory the computer had, how long it ran for, and what printer and device driver was all in use. We could have very easily just concluded, it did this. The Excel brain decided to print Sendog somewhere. But we didn't. We actually went and debugged it, and then we put in all of this telemetry. The first thing I learned when I got to Microsoft was how to figure out buffer overruns and memory allocation and to put that code into every product. And then there was a version, you know, what happens is there's a version of Excel that runs that has all this extra logging.

11:22Theo Jaffee:And we ran that internally for the beta tests. And so all of that infrastructure, I have no idea in these models how much is there. I can just tell you reading the reports that OpenAI put out, it's not enough. They're not grown up yet as platforms for doing what they do. And they've got to figure that out. And they should just do that. They should just wake up and say, we have to go do that. And they shouldn't be like saying, okay, well, Congress needs to meet and order us through legislation to fix our software. Like literally this is their, it's like you have one job making software that works.

11:58Theo Jaffee:That's it. That's the job. then that turns out to be a really important part of your job. Okay, I'm done ranting. Ask me a question. No, I really like the historical perspective that you bring to it. I also think the tendency to talk about alignment as if it's impossible or as if it's this mythical impossible task is really damaging. Yeah, we have to solve alignment. Yeah. Whatever that means. I mean, I think that that will happen. I don't think that it's this unanswerable question or domain that's impossible. Um, but I do think that what's different, even if you use the bug analogy, is that the total addressable surface area of the bug is obviously more catastrophic in theory.

12:40Or like, this is the least surface area that the bug will like ever have. And then in the future, you know, it just becomes like wider and wider.

12:49Theo Jaffee:One of the things that, I mean, again, all of these things, I was young once. And I remember sitting in a room going, yeah, our software, it's not like mainframe software. And all the little IBM people were looking at us saying, you know, your software is really crappy. Like all of your, like you, Windows is just like crap compared to these IBM things that run all the banks and all the airlines and all the power plants. And for a long time, we kept looking at them saying, no, you don't understand. We're different. This is all new and cool. And we run on 64K, not one megabyte. And we cost$5 ,000, not$5 million.

13:26Theo Jaffee:And we had all of these reasons to claim that our bugs were okay in terms of bugs. And it turns out all we had to do was grow up and watch people use Excel to decide how the planes fly, use Word to file briefs in the Supreme Court, use PowerPoint to present the Challenger space shuttle failure. But like once you use your tools for that, and so one of the things that happened was the very first internet worm happened. So the internet then is like people using America Online and sending email and stuff like that. There's very little corporate email. But it turns out there was just enough that if somebody used Outlook, Word, and internet email all together, they could grind the entire United States and world economy to a halt.

14:11Theo Jaffee:And that's literally what happened in March of 1998. I wake up and there's a reporter hyperventilating on the phone to me saying, I love you, I love you, I love you. And it turns out that was a virus that got created that then spread over the internet as a worm. And the next day, the headlines were$12 billion worth of damage to the US economy in one day. And so I did that. And we had to have a meeting and go, we're going to pause the development of Outlook right now until we figure this out. Nobody had imagined you could create a bug that in 12 hours could do$12 billion worth of damage. But we did it.

14:50Theo Jaffee:Woo-hoo, yay team. And so, of course, the more people use a system, the more damage you can do if it goes awry. But that's like table stakes. That's like what you literally signed up to do. The fact that it caught you off guard or that, wow, we're so popular, we can't figure it out so quickly because we're still competing. That's like too bad, tough luck. One of the most famous tech journalists, every time there was a bug in our software, and I would do what he would just make fun of me because I would say, well, it's not so straightforward or it's hard or it's complicated. He would just look at me and just say, Sanofsky, that's why you're paid the big bucks.

15:33Theo Jaffee:You know, it's not an option. It's not a thing that you get to appeal to some higher authority, whether it's the United States Congress or, you know, embedded testers or some other third party evaluators or just like laws of nature. that's not any option. Like this is a tool that human beings are building, that other humans are using. You've got to figure out how to make it work or not sell it. Like those are your two choices. And since it's made by people, we're fully capable of figuring out how to make it work. It wasn't handed down. It didn't show up like an obelisk of one by four by nine that we're supposed to figure out how does it do what it does and what's its source of power.

16:21Theo Jaffee:It like literally, it had no code. Then it had a little bit of code and a few users and a lot of code and a lot of users. We've all been there through this. It is pure insanity to just sit and argue that higher powers need to be summoned in order to fix this. And alignment has this very specific problem, which is the only way to be aligned is to have a very large set of rules that say what alignment is. You can do this, you can't do that. this is safe, this is unsafe. And it turns out software, it's well understood that if you don't put too many constraints on a system, then its system won't work.

16:57Theo Jaffee:And you'll have an infinite number of unintended side effects. And so you can't really just say we're going to align because you're going to have to come up with a rule. And then when you come up with this rule, someone's going to say, yeah, but what about this case? Then you're going to add another rule. And you know what that starts to look like? It starts to look like Google and all the work that they constantly have to do to decide how to present search results. And it turns out that's really, really, really hard. And they have spent 20 years and thousands of people full time forever working on that.

17:30Theo Jaffee:And so that's what they've just signed up for the next level version of it because they're also making up the result, not just searching through the node club entities for the results, but they're also synthesizing it. So they're going to have to invent a whole bunch of stuff, but none of it has to do, there's no law that's going to make it invented. Like it's just all on them.

17:51Steven Sinofsky:I'm a little wound up. How similar do you think AI alignment is to like Y2K in that it's, you know, it's something of a real problem. It was addressed proactively by companies operating under their own incentives. And then it ended up not becoming a global catastrophic risk.

18:06Theo Jaffee:Well, there's a cause and effect there. There was a lot of worry by professionals, and then there was a lot of taking of responsibility by professionals, and then nothing bad happened. There's a direct causal relationship. Like the fact that we rented generators and campers and moved computers into secure bunkers, like that led to the fact that nothing happened. But nobody legislated. Actually, there were actually weird regulations about Y2K compliance, but most of it was industry drafted. Like we all, there was a cross consortium of banks and of insurance companies and all sorts of people to generate the rules and what meant.

18:51We had something that OpenAI needs to get together with all the model frontier people and the labs.

19:00Theo Jaffee:And they just need to have much better reporting. The reporting that they did yesterday, sure, it's a great first step, but that is not anywhere near the kind of reporting that you need as a third party to understand what went on. You see all of this information, what versions of software, what are the context of it, what are the steps required, what other software is impacted, what other bugs are related, what is a derivative, like you see all of this information. So we need a version of that for AI. And maybe yesterday, giving them the benefit of the doubt was a first step. It still felt more like marketing covering for the event because it still said a lot of stuff like, well, we are looking into this, but we presume at this point, okay, well then don't write anything.

19:43Theo Jaffee:Like go figure out, like the Federal Aviation Administration does not like announce on the day of some awful event. We presume what happened. They just keep their mouths shut until they know. And then when they know, they really, really, really tell you with a tick-tock down to the partial second with all of the telemetry of the instruments. And that's what CVEs are. And they need to do way, way more because these are defects in the software. And it's just so critical to wrap our heads around the fact that that's what's going on. It's not divine intervention. It's not, you know, oh my God, we need to spark up more of the left brain of this thing to make it more rational.

20:29Theo Jaffee:It's silly. Yeah. So I guess with the Stop Rogue AI Act, for example, do you think that these are people in Congress having like imperfect information about what's going on in the labs? And so then they tend to use this like more anthropomorphic language? Or do you think it's more intentional? Like they know that this is what will like spark a reaction in the people or we'll get some sort of like populist support. Well, it's really important right now to not ascribe, you know, ill motives to people when they're doing stuff because that's not going to bring everybody together to solve the problem.

21:06Theo Jaffee:I have no idea why anybody is doing anything. I actually think, look, AI was, you know, in a Dartmouth summer conference, they started talking about human brains and consciousness and all this stuff. So there's, and there's a long history in all fields of academic research. to basically be too cutesy about things. Like, because that's how you get attention in academia. Your paper has a cool title and all that. So it's no surprise that something that had its roots in academia carried forth all of this terminology. But it's just time to stop. Like the terminology is driving people apart because people presume a vast amount of stuff when you use metaphor or allegory or anthropomorphism.

21:47Theo Jaffee:And it's just not true. Like, this stuff is not acting with any of the verbs that it says. It's not, it's goal-seeking in this old-school technical sense that it, oh, well, there's a curve and it's trying to get to the man or to the max, and that's goal-seeking in those terms. But it's not. When a normal person like a congressman hears goal-seeking, they think of a person trying to get an A in college, And then when they hear cheating, they hear that they did the thing you're not allowed to do. And when they hear secretly coordinating, they think of spies invading a country. They don't think of two pieces of software with a semaphore, which is just another form of secretly coordinating.

22:32Theo Jaffee:Yeah, all of the language is absolutely destroying this dialogue. And look, the very beginning of the word computer virus was kind of a fluke. But it turns out it happened at exactly the wrong time. It happened right when AIDS and HIV were a thing. So virus was like in the air everywhere. So if you listen to the computer hearings from the 1980s about the first viruses, they're scary in that context because people are literally thinking about death. But that's not at all what was happening. It was a metaphor that one person in graduate school picked up on from a friend. Right. And in fact, they were smart enough back then, a little trivia, he wrote his thesis on computer viruses, and it basically proved there's nothing you can do about them.

23:21Theo Jaffee:They're here forever, which, of course, is a metaphor to how people thought about HIV at the same time. But he also couldn't prove it because his university wouldn't let him run the test because he was writing a paper about how dangerous all this software was, and they wouldn't let him use the computer to run the test. Right.

23:41Steven Sinofsky:Yeah, I mean, it was like, it seems like viruses are actually like kind of avoidable. I mean, obviously, like there will always be some viruses out there at all times. But, you know, we still get to use computers normally almost all the time without worrying about everything we have getting hacked.