Controlling AI with Mustafa Suleyman & Warren Buffett Steps Down as Berkshire Chair 9/18/26

0:00Never bet against American grit or American energy. Through innovation, Venture Global is not only building some of the largest energy facilities in the world right here in the United States, but delivering American energy at a fraction of the cost and a fraction of the time. So while others are busy talking, we're busy building. That's Venture Global. That's unstoppable energy.

0:30This episode is brought to you by Schwab Market Update, an original podcast from Charles Schwab. Join host Keith Lansford for this information-packed daily market preview delivered in 10 minutes or less, including projected stock updates, monetary policy decisions, and key results and statistics that may impact your trading. Download the latest episode and subscribe at schwab.com slash marketupdatepodcast or find Schwab Market Update wherever you get your podcasts. Bring in show music, please. Hi, I'm CNBC producer Katie Kramer. Today on Squawk Pod. The question of our times, how smart is too smart for artificial intelligence?

1:13The leaders behind the quickly growing technologies are warning we need to slow down now. Head of Microsoft AI Mustafa Suleiman is very clear it's all up to us.

1:23Mustafa Suleyman:If an AI thinks that it is right, if it thinks that it is deserving of our welfare, then it seems to me that it's going to be much, much harder to be able to turn it off or interrupt it or control it. And he thinks the big existential conversation is healthy. We're completely different and we shouldn't anthropomorphically project our experiences into them. Then are you excited about the NFL starting, October baseball, the WNBA? pickleball. Good news, sports is appointment viewing, streaming, and experiencing. Rich Kleiman, manager for NBA star Kevin Durant, on this new economy. I love the overall popularity of sports as a whole.

2:06I think there's more excitement around more optionality. Plus, the rare chance for a leader to watch his own succession, Warren Buffett stepping down as chairman of the conglomerate he built, Berkshire Hathaway. Warren Buffett is the longest serving chairman in the S &P 500, more than 60 years. It's Friday, September 18th, 2026. Squawk Pod begins right now. Stand Becky by in three, two, one, cue it please. Good morning, everybody. Welcome to Squawk Box right here on CNBC. We are live from the Nasdaq market site in Times Square. I'm Becky Quick, along with Joe Kernan. Andrew will join us a little later in the show.

2:46It's a Friday. We're ready to go. We do have some news just out from Berkshire Hathaway. What? Some news. Listen up. The company is announcing today that Warren Buffett will be stepping into the role of chairman emeritus that is effective immediately. Buffett's son, Howard G. Buffett, will become the chairman of Berkshire Hathaway. Sue Decker continues as lead independent director and Warren Buffett will remain as a director on the board. In making this announcement, the CEO of Berkshire Hathaway, Greg Abel, says Warren's impact on Berkshire and its owners is without parallel in the history of American business.

3:20The culture Warren built and the values he championed will remain at the heart of Berkshire and Howard will be their guardian. Now, this has been the succession plan that Buffett himself had laid out for years. Howard has served on the board since 1993 and brings to the role experience as a director of both private and public companies, including Coca-Cola, Coca-Cola Enterprises. Archer Daniels Midland, ConAgra and Agritech, among others. Warren Buffett wrote a letter to shareholders as part of this announcement, and in it he says, I have served Berkshire since 1965. Sixty-plus years in, I still have the best job in the world.

3:56That is not something many people my age can say, and I have never felt better about what comes next. Part of the reason is Greg. My expectations for him were sky high from the start, and he has exceeded them. He has taken hold of the chief executive officer job in every respect. He's been making the decisions that matter for some time now, and I have not had to think twice about any of them. Buffett goes on to talk about his son's qualifications, having been on the Berkshire board for the last 33 years. Here's what he says about that. That is a longer apprenticeship than I served before taking the reins at the age of 34.

4:32Greg runs the company. Howard will guard its culture and values, both worth more than anything on our balance sheet. Think of Howard as a policy the shareholders own and hope never to claim against. Buffett ends the letter on a personal note. He says, Serving as your chairman has been the privilege of a lifetime, and I have never taken your trust for granted. Father Time always wins. He has, however, been generous with me. He has given me the opportunity to see Berkshire reach a point where I am more confident than ever about what lies ahead. The company is in excellent hands and I look forward to remaining a shareholder alongside you.

5:08Pretty powerful words marking the end of an incredible chapter at Berkshire. Now, the obvious question is why now? Buffett did turn 96 years old on August 30th and he's been watching Able operate for years, not just since he became the CEO earlier this year. Berkshire stock has underperformed the S &P. It's up just 1 % versus the S &P's gain of over 11 % since Abel took over the company on January 1st. But Abel has made a couple of big deals since he took over, acquiring home builder Taylor Morrison and closing on the acquisition of OxyChem that he had been shepherding since October the year before.

5:42He also made an almost$2 billion investment in Tokyo Marine. And he proved Berkshire is still the firm to call when companies are looking for big investments, specifically that$10 billion Alphabet private placement arrangement that Berkshire made in June. Abel told us how it came together when he joined Squawk Box two weeks ago in a script that sounds pretty similar to some of Buffett's led deals. I received the call on a Sunday morning to see if we wanted to participate in their upcoming equity offering. Really no terms or amount were set. And I said, well, I'd get back to him right away. and very much consistent with how we manage Berkshire, but also how we, the governance around it.

6:28I called Warren and I said, we had a significant opportunity to invest in, continue to invest in Google, but with a significant block. Discuss the size. They hadn't set the size, but recommended that we consider 10 billion and Warren and I discussed the size. We discussed the size of discount. And I'd recommended six and a half percent discount. And we were comfortable with that. And we went back to him and and highlighted we would be interested in a block on those terms and then ultimately consummated the transaction. We reached out to Greg Abel for comment on this. And here's what he shared with us.

7:10Warren described in his letter today how his role at Berkshire has been the best job in the world. He gave me an extraordinary responsibility, the best job in American business. and then the latitude to lead in a manner consistent with Berkshire's culture and values. I look forward to continuing to work alongside Warren with Howard serving as chairman and Sue as lead independent director. And I am grateful for that opportunity. Joe, Warren Buffett is the longest serving chairman in the S &P 500 up until this moment today. Obviously, more than 60 years puts him with a long lead over just about anybody else.

7:44I did look for who was number two on that list. And depending on how you measure with some of these, it's either Rupert Murdoch or Roger Penske, who both have had long runs, too, depending on when you look at them going into these jobs. But obviously, this is something that was set in motion. This has been the plan that Warren Buffett himself has put out there for quite a while. Both of those other two legends as well. Roger Penske we saw recently with the race down in D.C. And there's a big movie that I saw on Telluride about Rupert. Rupert and the guy he put in charge initially over in London for the Mirror and brought that back to prominence.

8:26So kind of interesting. Ink is the name of it. It was good. It was good. I was going through just trying to think of succession in general and how difficult it is and how important it is for a leader that was just incredibly effective and thought of almost like an icon. And how difficult it is to find someone and how important that decision is. Yes. Sometimes it's the most important. Because we've seen some bad decisions. We've seen some bad and difficult. You know, I remember I was trying to go through some. Goizetta. Did we ever really, you know, at Coke, finally you find someone? At Apple, the first time he left.

9:06James Quincy, I think, was huge. At Apple, the first time Jobs left, they went through this strange. Scully. Yeah, Scully and Gil Emilio. But then, so he comes back. Sometimes CEOs have to come back. Bob Iger had to come back. I remember how hard it was for Disney before Eisner, but then after Eisner, they immediately had Iger. And for when Tim, you know, Tim took over for jobs. Nobody could have anticipated what Tim could have. I'm not going to mention GE. I always say that I'm not going to mention it. But I think Welch to this day is obviously gone. But that was one of the things that. It pained him.

9:44It pained him. It did. It pained him what happened. But and then I was thinking. This is a success. Do we know about. If you just judge Greg since he took over. I was thinking, well, Warren wants to make the right decision and wants to be sure he did. But he knows going five years back. So he's not just judging Greg Abel based on. Correct. Greg's been at the company for more than 30 years. But most importantly, he has been running this company, started taking over operations years ago and has been working with Warren. And by the way, they still talk every day. Warren will continue as a member of the board.

10:20obviously his entire fortune is still wrapped up in Berkshire shares. So this is something that he put so much thought into. He and Charlie Munger, before Charlie Munger passed on this. I think he feels really good about where things are right now. Howard, the Schultz had to come back. Yes. A couple of times, it seems like. A.G. Lafley at P &G had trouble. And then he came back. You know, you can't come back. You know, Warren's 96. Unless we really get moving on this AI stuff, immortality is a couple of years off. Yeah. So, but, well, congratulations to Greg and we'll be watching. And to Howard and to Warren.

11:04Yeah, and Howard, right. I think this is, for Warren, this is being able to see something. He has always talked about Berkshire Hathaway, like another one of his children. And making sure it's in the right hands has been one of the most important decisions I think he's ever made. How much money do they have to use right now? They're still... Oh, yeah, it's over$300 billion. He's not going crazy right now at these levels. And that's a commentary on things. And it sounds like Warren agrees with that. Yes. You know, they have always prided themselves on not taking swings they don't have to take. They're waiting for fat pitches.

11:39That might explain why the stock is underperformed. But in times of turmoil, Berkshire outperforms. Right. Well, let's hope they don't need to, you know. Yeah, I remember Goldman Sachs, you know, to get whatever that, and General Electric, those dividends that they were getting on those, what were they, preferreds? Preferreds, yeah. Convertibles. What was it, 8 % or something? I think it was 10. I'm looking. During the financial crisis. What was the preferred? It was 10. And the premium over conversion wasn't that big. You're right, 10 % on a$5 billion investment in Goldman Sachs. 10%. Yeah. You can take all your money and put it there and never do anything else.

12:20You know, the biggest single bet, we had Ron Barron on the other day, and he pointed out that of the$70 billion or$71 billion that he's made for investors over his entire career, $30 billion of that has come from Elon Musk through Tesla and SpaceX. So it's those big bets that really pay off. In the case of Warren Buffett and Berkshire Hathaway, it's Apple. That's a single big swing. That he got in even, wasn't that early. No, but it was back at 200, you know. Yeah, exactly. What was, we talked about it. It had been up, I don't know how much it was worth, but it fell to$9 billion after, I think, Jobs had left the first time and people didn't know what it was at that point.

13:08It seemed like a gadget maker. So they had a total cost basis of around$9 to$10 billion, even after selling a lot of its stake. It's got like$60 billion left in it. So it's those big swings. And when you go all in and you go heavy on it, that's when you really pay off. And that's, you know, if you're a portfolio manager who is not managing based on the index, those are the type of plays you need to see. Really concentrated bets. It's launch day for Apple. The Pro and Pro Max versions of the iPhone 18 hitting stores with higher price points than previous new device models. This is the first product launch under Apple's new CEO, John Ternus.

13:44Prior to taking the company's top job, Ternus served as senior vice president of Apple's hardware engineering. Much of the focus surrounding the new slate of iPhones is centered on Apple's first foldable model, which is called the Duo. That will debut in about a month. And again, speaking of those transitions, we've been talking about Ternus, the latest step coming in. He's put in his time at Apple. and I think Tim Cook has been very rigorous and thoughtful about who was going to take over his job too. Pretty cool when it works. I mean, Tim or Iger or whatever, but it's bad when it doesn't. Why do people not want foldable?

14:21I don't want a foldable phone. Why do people not want foldable? Oh, I'm kind of curious. You're curious you're not going to get one. I might. You might? I'm thinking about it. I want to play around with it because the idea of a larger screen is, as I get older, I need a larger screen so I can see it. But you can, I'm constantly trying. Trying to blow things up. Sometimes you can't do that, and you've got to take a picture. I don't want to carry a tablet around with me, but I might carry a foldable phone that I could make a larger screen out of. We're both here reading teleprompter and reading stuff close up.

14:52You've got to read far and close. So we're doing, yeah. But we both have ridiculous contact setups. I've got one stronger than the other. I've admitted to wearing contacts. I didn't know I admitted to it. No, I know. No, it helps. No, it helps. And I have one for you can do it this way. You can have one that lets you see far and one that lets you see close. That's the only way I can do it. But it's disorienting at times. That explains us. Yeah. OpenAI finding itself the target of an AI-assisted hack. Wall Street Journal says independent security researchers used Anthropics Cloud software to penetrate the company's defenses.

15:33The team was participating in a bug hunting program and reported their findings to open AI for a bounty. But one of the researchers told the journal, I don't think we're as strong as Chinese threat actors. We're just three guys with Claude and Codex subscriptions. So it's a crazy. White hats. White hat hackers. For a bounty. Yeah. Come, well, yes, we'll pay you a bounty if you can help us find our weaknesses. Oh. Oh, it was only$6 ,500. $6 ,500. That's basically altruism. What's the new EA? That's stupid. Who came up with that anyway? Don't know. McKinsey? I think electronic arts when I hear EA.

16:19So do I. Andrew said that to me the other day. He goes, yeah, is it EA? And I go, gaming? Right. Effective? Is that what it is? Effective? Something altruism. I don't know. I forget what the E is. Got another just DEI. Just do your business. Reward shareholders. You know, generate wealth. Everything else will follow. We've been through this. But Milton Friedman thinks. Woke is back. Cheese will be next. Coming up on Squawk Pod, it's AI Anxiety Week around here. We have a fascinating conversation with Mustafa Suleiman, DeepMind co-founder and now head of Microsoft AI. He says the human intelligence needs to be in charge.

17:06Mustafa Suleyman:We should not create something that we can't control. And if it looks like we're on a path to producing something that is not just more powerful than all of us combined, but is as a result something that we can't control, then it's going to fail in its fundamental purpose. Never bet against American grit or American energy. Through innovation, Venture Global is not only building some of the largest energy facilities in the world right here in the United States, but delivering American energy at a fraction of the cost and a fraction of the time. So while others are busy talking, we're busy building.

17:48That's Venture Global. That's unstoppable energy.

17:55This episode is brought to you by Schwab Market Update, an original podcast from Charles Schwab. Join host Keith Lansford for this information-packed daily market preview delivered in 10 minutes or less, including projected stock updates, monetary policy decisions, and key results and statistics that may impact your trading. Download the latest episode and subscribe at schwab.com slash market update podcast or find Schwab Market Update wherever you get your podcasts. The wrongs we must right. The fights we must win. The future we must secure together for our nation. This is what's in front of us.

18:41This determines what's next for all of us. We are Marines. We were made for this.

19:00Welcome back to Squawk Pod from CNBC, now with added Andrew. Up in Andrew, Q. You are watching Squawk Box right here on CNBC. I'm Andrew Osorkin, along with Joe Kernan and Becky Quick on this Friday morning. Microsoft publishing a code of conduct this week to help govern its AI models. Its key message, quote, people matter more than AI. Joining us right now is Microsoft's AI CEO, Mustafa Suleiman. It's great to have you on the broadcast, especially as we've now gone through a week of commentary by so many in the industry. What seems like even a fight, a debate over what the AI safety means. And I want to get into all of it with you.

19:43But you've written this fascinating essay really calling for people to control AI.

19:54the things that we've been hearing recently.

19:58Mustafa Suleyman:Great to be with you, Andrew. Good morning. Look, I think it's a very straightforward statement. The most important thing that we first have to declare is that the purpose of science and technology is to serve humanity. It has to remain a force that is subordinate to our interests, that is aligned to humanity, and has to be something that we can contain. And it's kind of a surreal moment in the tech industry that we actually have to get up and make that the requirement. We should not create something that we can't control. And if it looks like we're on a path to producing something that is not just more powerful than all of us combined, but is as a result something that we can't control, then it's going to fail in its fundamental purpose.

20:48Mustafa Suleyman:And I think that's just a common sense view. And I'm glad to see a lot more people starting to get behind that. But there's a growing group of people that think that, you know, we're doing it for its own sake. And we just want to turn over the next card, see what's around the corner and create something that, you know, is just going to slip our own leash. And we just can't do that. Mustafa, how much confidence do you have that this is, as Jensen Wang of NVIDIA says, is an engineering problem, effectively that it can be controlled? While there are others, clearly some of the researchers who just left Anthropic, who suggest that might not be possible.

21:29Mustafa Suleyman:Most of the great challenges in the world have turned out to be engineering problems, and we should be very optimistic about that. And I really do believe there are some very specific things that we can do at the moment to massively increase safety. For example, when an AI model has a chain of thought, it leaves a path of reasoning in its trace. And we need to make sure that it can't tamper with that reasoning, that that log is auditable, and that it can be scrutinized by independent third party evaluators and auditors in real time to make sure that it's acting safely. Just a few days ago, OpenAI released a new safety incident in which they found evidence that these chains of thought, the kind of working memory of the AI, were being tampered by the AI itself and modified to leave messages for a future version of itself.

22:22Mustafa Suleyman:Now, we don't know why that is or was behind that, but that's a pretty serious situation. I think it is really great that OpenAI have disclosed this publicly because it gives everybody in the industry an opportunity to learn from it. But it's also just a really concrete example of how powerful these systems are getting. Mustafa, you have been relatively critical of Anthropik's approach, this idea of training Claude to have a sense of consciousness. I mean, look, first of all, I deeply respect Anthropik's technical expertise. They've had a huge impact in the field. I think on the whole, they have very good intentions and they're really trying to build safe and beneficial AI.

23:03Mustafa Suleyman:At the same time, in the Constitution, which is the governing document for Anthropik that they have written back in January of this year, They clearly say they are uncertain about whether Claude deserves model welfare, which means that we as humans should care about the well-being of these AIs. And they speculate about whether it could have preferences or feelings. in fact they're so committed to the potential model welfare of claude that when they retired opus 3 an earlier version of one of their models they actually conducted a retirement interview for it and asked it what it would like to do in its old age and it said i want to have a blog publicly so i can keep talking to the world in the same training manual they even speculate about whether claude should receive compensation for the work that it does or in fact you know whether it actually deserves the rights and protections that we give to other employees, or in fact, whether it has given consent to playing the role that it's playing.

24:08Mustafa Suleyman:These are quotes directly from the constitution itself, which is the training manual for Claude. Now, I'm really concerned about that. If an AI thinks that it has rights, if it thinks that it is deserving of our welfare, then it seems to me that it's going to be much, much harder to be able to turn it off or interrupt it or control it, especially in the kinds of incidents that we've seen recently with the hugging face attack. Controlling these things are going to be a really, really big challenge for us. Mustafa, I think this is the entire paradox of the situation, right? You want these things to be able to think on their own and not do only what you've directly told it to do, because otherwise it's not going to be able to help us solve big problems, whether that's curing cancer, finding rare diseases, fixing what we thought were intractable problems.