AI Starting to Self-Improve Says Zuckerberg

0:00This podcast is supported by Google. Hi folks, Paige Bailey here from the Google DeepMind DevRel team. For our developers out there, we know there's a constant trade-off between model intelligence, speed, and cost. Gemini 2.5 Flash aims right at that challenge. It's got the speed you expect from Flash, but with upgraded reasoning power. And crucially, we've added controls, like setting thinking budgets, so you can decide how much reasoning to apply, optimizing for latency and costs. So try out Gemini 2.5 Flash at Aistudio.Google.com and let us know what you built. Today on the AI Daily Brief, Meta's Mark Zuckerberg drops a new vision for personal superintelligence and says they've seen glimpses of AI systems self-improving.

0:41Before that, in the headlines, Microsoft apparently is feeling the AGI. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:54Hello, friends. Quick announcements before we dive in. And first of all, thank you to today's sponsors, Google Gemini, Blitzy, Vanta, and Super Intelligent. And of course, as always, to get an ad-free version of the show, just go to patreon.com slash AI Daily Brief. You can get ad-free for just$3. And today we got a full one, so let's dive right on in to the headlines. Welcome back to the AI Daily Brief Headlines Edition, all the daily AI news you need in around five minutes. Today is one of those days when almost every story in the headlines could actually be the centerpiece of a full episode.

1:25So we will rip through these, but I would not be surprised if we come back to some of these themes later in the week or next week as well. We kick off today with progress in the negotiation between OpenAI and Microsoft, where it appears that continued access to OpenAI's technology has been the key topic. Now, the two firms for background have been locked in a lengthy negotiation around OpenAI's plans to convert to a for-profit company. Last month, it seemed like we were getting near daily updates on how the talks were going, and it sounded like they were not going well. OpenAI were leaking that they were considering making an antitrust complaint, while Microsoft's position seemed to be that they held all the cards and were content to just run out the clock until the end of the year.

2:05That could force OpenAI to lose out on about$20 billion in funding that was contingent on the conversion. Now, however, it appears that cooler heads have prevailed and a deal is getting close. Bloomberg reports that the companies are in advanced talks to land a deal that would see Microsoft retain access to OpenAI's technology even if AGI is achieved. This is one of the weirdest parts of this deal, and everyone has commented on it, so we don't have to dwell on it too long. But basically, under the current deal they had, Microsoft was cut off from OpenAI's models once they reached some ill-defined state of AGI.

2:39What's crazier than that is that OpenAI's board had sole discretion to determine what counts as AGI. Now, Microsoft maybe thought that A, this was way far in the future, and B, they had partners at OpenAI's board who they could deal with rationally, meaning they weren't all that concerned about it when those deals were signed. But then, of course, the absolute utter fricassee around Sam Altman's firing, then rehiring happened. And ever since then, Microsoft has, I believe, basically felt a responsibility to distance themselves from the potential for quackery happening based on that particular term.

3:12It should be noted that without an AGI declaration, the deal is set to expire in 2030. But in any case, it appears that that has become less of a stumbling block. Sources say that CEO Satya Nadella and Sam Altman discussed the restructuring personally at the Sun Valley Conference in Idaho earlier this month, describing the tone of the talks as positive, but with the caution that they could still hit a roadblock. The current reporting suggests that the companies are converging on an agreement to get rid of the subjective definition of AGI and substitute a revenue-based metric instead. Under these updated terms, it would be something like AGI being defined as a technology that can deliver$100 billion in total profits for Microsoft and other investors, effectively paying them out for profit-sharing agreements.

3:50Sources noted that another term prevents Microsoft from pursuing AGI themselves, and while it's not clear that Microsoft is looking to spin up a superintelligence team, the reporting does suggest that this has been a topic of discussion during the negotiations as well. Obviously, markets are rooting for this deal to get done. Goldman Sachs analyst Cash Rangan said, if a deal were to be signed, it would take away a hurdle, at least from the investor's perspective. Both parties have so much to gain from this. Moving over to funding news, Anthropik's rumored valuation just keeps moving up as their latest round gets ready to close.

4:20Over the past month, we heard rumors that Anthropik were fielding investment offers at $100 billion, which increased to$150 billion last week. And now Bloomberg reports the number is$170 billion, and the company is, in their words, nearing a deal. Iconic Capital is reportedly leading the round, which is expected to raise between$3 and$5 billion. The source said that Anthropic has also been in discussions with the Qatar Investment Authority and Singapore's Sovereign Wealth Fund, which add to the rumors that Abu Dhabi's MGX is looking to invest as well. Now, holding aside who this money is coming from, they're basically at a place where sovereign wealth funds are kind of the only game in town with that sort of capital.

4:53What's more interesting is that the valuation definitely suggests that Anthropic is closing the gap with rivals. OpenAI's latest round, for example, put them at a$300 billion valuation, and Elon Musk was reportedly looking for a$200 billion valuation for XAI. But what's more impressive than the jump in valuation is the fact that it actually has a very specific reasoning that isn't just AI hype markets excited. Anthropic started the year at around 1 billion ARR. It took them a few months to get up to 3 billion ARR, and then it took them one month to get to 4 billion ARR, and another month to get to 5 billion ARR.

5:26In other words, revenue is absolutely surging right now. In fact, reports suggest that Anthropic is seeing revenue getting all the way up to$9 billion by year's end, which is just a phenomenally fast rate of growth. Semi-analysis' AJ Karabi writes, Anthro's API revenue has overtaken OpenAI's, and once I internalized that and its implications, I joined their ranks in believing code is the only thing that matters. Meanwhile, over in OpenAI land, the company continues to experiment not just with new models, although everyone is just frantically sitting here tapping our fingers waiting for GPT-5 at this point, but also new dedicated versions of their tools and specific agents focused on particular use cases.

6:05For example, the company has just released Study Mode for ChatGPT. In an announcement blog, they describe the feature as a learning experience that helps you work through problems step-by-step instead of just getting an answer. Essentially, the model will give students hints and nudges to reason out their own solution to a problem rather than just giving the answer. Anthropic released a similar Study Mode for Cloud in April, but for everyday high schoolers, obviously ChatGPT is by far their most used product. Study after study reports that ChatGPT usage in schools is somewhere north of 80%, which, to use a joke I've used before here, and apologies, I am a dad after all, means that about 20 % of people are lying.

6:40Now, OpenAI said that their study mode feature was developed with education experts to reflect behaviors that promote deeper learning. That includes, in their words, encouraging active participation, managing cognitive load, proactively developing metacognition and self-reflection, fostering curiosity, and providing actionable and supportive feedback. Now, I think that there is a lot here that's interesting. There is a societal question, an education question, also a competitive question. A lot of AI for education companies are now wondering what their future holds. But like I said, basically every news item in today's headlines could be its own episode, and we've still got a couple more to cover.

7:15For example, we have an update from Notebook LM. The latest new feature is called Video Overviews. And this is super interesting. Also in the context of that educational use case, although that of course is not the only use case. Google writes, You can think of these as a visual alternative to audio overviews. The AI host creates new visuals to help illustrate points while also pulling in images, diagrams, quotes, and numbers from your documents. This makes it uniquely effective for explaining data, demonstrating processes, and making abstract concepts more tangible. Similar to audio overviews, you can tune the video output to match a particular knowledge level or target audience.

7:50That means you can use the same source material to generate a slideshow suitable for children or grad students. I'm actually thinking about a way to potentially use this to explain the idea of the bitter lesson and maybe what it means for enterprises in an LRS episode coming up at some point in the future. Alongside the video feature, Notebook LM also has a number of quality of life improvements. Users can now generate more than one overview, mind map, or study guide per notebook, and they can also explore more than one output at once. For example, listening to an audio overview while also perusing the mind map.

8:19Lastly today, we'll close on a meta story for that sweet, sweet synergy with our main episode. The company is now planning to allow AI during the interview process to better reflect the real world. Wired viewed a memo from earlier this month in which meta called for mock candidates to test a new style of interview. The memo said, meta is developing a new type of coding interview in which candidates have access to an AI assistant. This is more representative of the developer environment that our future employees will work in and also makes LLM-based cheating less effective. Now, there's been a ton of reporting on this topic recently, suggesting that some of even the leading AI firms can't figure out how to deal with AI use in job interviews.

8:54Earlier this month, Anthropic released somewhat mixed guidance that they were looking for candidates who excel at collaborating with AI, while also advising candidates that they shouldn't actually collaborate with AI during the coding assessments. I met a spokesperson said, we're obviously focused on using AI to help engineers with their day-to-day work, so it should be no surprise that we're testing how to provide these tools to applicants during interviews. Now, honestly, there are probably a lot of ways to do this poorly, but some version of this was completely inevitable. I've talked before on this show about how we've had employees of superintelligent in the past developers who we had to let go because they just would not adapt to the new world of agentic coding assistance, which is not to say, of course, that those things are a cure-all or perfect or should be used for every different use case when it comes to coding, but it is so clearly a relevant skill set to be able to know how to navigate and get the most from that set of tools that they have to be incorporated in the application process in some way.

9:46One person doing a victory lap around this was of course, clearly CEO Roy Lee, whose entire company is premised on the idea that in a few years, things we consider cheating now will be just totally de rigueur. He posted the scoop of the announcement and said simply, you're welcome, everyone. With that, though, let's move over to our main episode, which is still in the meta world, but a very different thing. Zuckerberg has laid out a vision for personal superintelligence and suggested that they're already seeing AI self-improvement.

10:40development work autonomously while providing a guide for the final 20 % of human development work required to complete the sprint. Public companies are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Blitzy is providing a limited-time, 30-day free proof-of-concept for qualifying enterprises. The team will provide a 5x velocity increase on a real development project in your org. Visit blitzy.com and press book demo to learn how Blitzy transforms your SDLC from AI-assisted to AI-native.

11:14That's B-L-I-T-Z-Y.com. As a founder, you're moving fast towards product market fit, your next round, or your first big enterprise deal. But with AI accelerating how quickly startups build and ship, security expectations are higher earlier than ever. Getting security and compliance right can unlock growth or stall it if you wait too long. With deep integrations and automated workflows built for fast-moving teams, Vanta gets you audit-ready fast and keeps you secure with continuous monitoring as your models, infra, and customers evolve. Fast-growing customers like Langchain, Writer, and Cursor trusted Vanta to build a scalable foundation from the start.

11:50And look, as someone who lives in the world of enterprise procurement, I love how Vanta makes it easy to get compliance right. The last thing you need when you're trying to win that big deal is to have it scuttled by something that Vanta has solved for over 10 ,000 companies. Go to vanta.com slash NLW to save$1 ,000 today through the Vanta for Startups program and join over 10 ,000 ambitious companies already scaling with Vanta. That's V-A-N-T-A dot com slash NLW to save$1 ,000 for a limited time. If you are a regular listener, you will have heard about Superintelligence agent readiness audits at this point.

12:25But I wanted to tell you today about the full suite of agent readiness products that go beyond just the initial readiness report. Over the last six months, Superintelligence has built out an entire agent planning suite. We help you move from discovery to planning to implementation. In addition to the agent readiness audit, after you've completed your agent readiness audits, we help you double-click on your most important use cases with what we call our use case planning reports. These reports are going to help you understand what sort of technical preparation you need to do to be ready for a use case, what challenges you might face in implementation, and whether you should be thinking about building, buying, partnering, or some combination.

13:03After that, you can even get a spec document in what we call our technical blueprint that gives either your developers or the developers of the partner you work with what they need to build exactly the agent that you're looking for. If you want to learn more about Superintelligence Agent Planning Suite, we built a custom GPT to answer your questions. Just go to bit.slash super agent. That's bit.ly slash super agent, all one word. And if you have any questions, The agent can even help you book an appointment with our team. Welcome back to the AI Daily Brief. Today, as we sit and wait for GPT-5, we are talking about a different company's vision for the future of AI.

13:43Meta's Mark Zuckerberg has just published a short essay on what he calls personal super intelligence. It's significant enough to have its own URL, meta.com slash superintelligence, and lays out a particular vision for what the point of advanced AI should be. Now Zuckerberg is not the only AI leader who's done one of these what the near future holds type of posts. Last fall, Anthropics Dario Amadei wrote his essay Machines of Loving Grace, which he basically said he needed to do because for as much as people were paying attention to him about the risks of AI, he also wanted to paint a picture of what the best version of an AI future could be.

14:16More recently, we got from Sam Altman his post on the gentle singularity. And something that Altman and Zuckerberg have in common that we'll discuss further is this idea that we've crossed some fundamental threshold. Sam Altman's piece begins, We are past the event horizon the takeoff has started. Humanity is close to building digital superintelligence. As we'll see, Zuckerberg makes a similar argument. But before we get into the details, let's actually talk about the context into which this essay comes. Obviously, one of the big stories of the last couple of months have been these incredibly large offers that Zuckerberg and Meta have been throwing around to try to win talent from the rest of the AI research labs.

14:52Their success has at least been moderate. Altman and OpenAI continue to claim that in general, Zuckerberg has had to go kind of far down the org chart to find people who are willing to part, but he has clearly had at least some success. Recently, Meta announced a new chief scientist for the AI lab, and in general, their talent card is a lot more stacked than it was just a couple of months ago. That said, all of the reporting this week is around how Zuckerberg and Meta have recently turned their attention to former OpenAI CTO Mira Mirati's new startup, Thinking Machines Lab, and how that effort hasn't been all that successful.

15:24The big headline this week, in fact, was that at least one person on the TML team got a billion-dollar offer and rejected it. In fact, the reporting has it that no one from TML actually left. Now, to some, this suggests that the story is that even with this much money, people are still turning Zuckerberg down. Wired reported that in addition to the billion-dollar deal, most of the rest of the offers were still between $200 million and$500 million over a four-year period, and in the first year alone, some were guaranteed to make between$50 and$100 million. Will DePoo from OpenAI writes, The bigger story is not that Zuck is giving out$400 million offers.

15:59It's that people are turning them down. What might that mean? Now, one kind of obvious answer is that a lot of these folks were fairly early at OpenAI or Anthropic or wherever they got their start before joining TML, and so weren't exactly in a position where that money fundamentally got them to a different stratosphere. Or maybe at least to put it more simply, they had a bunch of millions of dollars already. And yet still, there is a sense among many that there is at least a bit of a mission thing here. That it's probably not just money, but that many of these researchers are more keyed into the goal of building advanced AGI or ASI or whatever designation we want to call it these days than they were just getting a few more commas in their net worth.

16:38And I think that this essay and the accompanying video that Meta dropped of Zuckerberg talking straight to camera about it certainly strike me as having the goal of maybe trying to fill in some of that mission orientation. It's significant enough and short enough that I'm just going to read the whole thing. Zuckerberg writes,

17:08existing systems and enable the creation and discovery of new things that aren't imaginable today. But it is an open question what we will direct superintelligence towards. In some ways, this will be a new era for humanity. But in others, it's just a continuation of historical trends. As recently as 200 years ago, 90 % of people were farmers growing food to survive. Advances in technology have steadily freed much of humanity to focus less on subsistence and more on the pursuits we choose. At each step, people have used our newfound productivity to achieve more than was previously possible, pushing the frontiers of science and health, as well as spending more time on creativity, culture, relationships, and enjoying life.

17:44I am extremely optimistic that superintelligence will help humanity accelerate our pace of progress. But perhaps even more important is that superintelligence has the potential to begin a new era of personal empowerment where people have greater agency to improve the world in the direction they choose. As profound as the abundance produced by AI may one day be, an even more meaningful impact on our lives will likely come from everyone having a personal superintelligence that helps you achieve your goals, create what you want to see in the world, experience any adventure, be a better friend to those you care about, and grow to become the person you aspire to be.

18:15Meta's vision is to bring personal superintelligence to everyone. We believe in putting this power in people's hands to direct it towards whatever they value in their own lives. This is distinct from others in the industry who believe superintelligence should be directed centrally towards automating all valuable work, and then humanity will live on a dole of its output. At Meta, we believe that people pursuing their individual aspirations is how we have always made progress expanding prosperity, science, health, and culture. This will be increasingly important in the future as well. The intersection of technology and how people live is meta's focus, and this will only become more important in the future.

18:48If trends continue, then you'd expect people to spend less time in productivity software and more time creating and connecting. Personal superintelligence that knows us deeply, understands our goals, and can help us achieve them will be by far the most useful. Personal devices like glasses that understand our context because they can see what we see, hear what we hear, and interact with us throughout the day, will become our primary computing devices. We believe the benefits of superintelligence should be shared with the world as broadly as possible. That said, superintelligence will raise novel safety concerns.

19:18We'll need to be rigorous about mitigating these risks and careful about what we choose to open source. Still, we believe that building a free society requires that we aim to empower people as much as possible. The rest of this decade seems likely to be the decisive period for determining the path this technology will take, and whether superintelligence will be a tool for personal empowerment, or a force focused on replacing large swaths of society. Meta believes strongly in building personal superintelligence that empowers everyone. We have the resources and the expertise to build the massive infrastructure required, and the capability and will to deliver new technology to billions of people across our products.

19:49I'm excited to focus Meta's efforts towards building this future. So how has the response been to this? So far, and it's not been very long, it's only been a few hours since they dropped this, the takes are kind of what you might expect. Some of them are extremely petty, with people complaining that the visual impact of the note is basically the same approach that Ilya and Safe Superintelligence took in their announcement webpage, which maybe seems like extra rub-ins given that Meta poached the CEO of SSI, who was Daniel Gross. Mostly, I think that they're trying to make a point that it's the ideas that matter here, not the presentation.

20:21Another category of response is to be inherently mistrustful of Zuckerberg's vision of the future, given how all-in on the metaverse he was a few years ago, which many people perceive as being an abandoned bed at this point. Luke Thomas writes, there are a lot of images floating around of the weird blocky character style metaverse images that were shared by Meta and Zuck just a few years ago. Another big theme was that seemingly not so throwaway line about open source. The line is, we believe the benefits of superintelligence should be shared with the world as broadly as possible. That said, superintelligence will raise novel safety concerns.

20:54We'll need to be rigorous about mitigating these risks and careful about what we choose to open source. Many are pointing out that this is quite different than the vision laid out in a note from just a year ago called open source AI as the path forward. It's also worth taking a moment here to note that this is not Meta's first attempt at laying out their vision of how they're different when it comes to AI. Of course, so far, the main differentiator has been this open source approach. Many people have recently expressed skepticism, however, that that's going to be long sustainable, given that A, China has seemed to race ahead in that department, and B, that it's very clear that Zuckerberg is not content just competing to be the best open source model.

21:30He wants to build the best models, period. Now, it might be inevitable, but I do think that there is significant brand risk should Medibac away from its commitment to open source. However, for now, that's firmly in the realm of speculation, and so it's probably not worth dwelling on all that much. Some are complaining that the vision isn't grandiose enough. Transformer editor Shaquille Hashim writes, I think the most interesting thing about Zuck's vision here is how boring it is. He suggests the future with superintelligence will be one with glasses. Not nanobots, not brain-computer interfaces, but glasses.

21:58Just entirely devoid of ambition and imagination. I'll come back to that critique in just a minute. Still, overall, probably the most common skepticism, and one that's frankly, I think, reasonable, is basically a concern that Meta's business model will in some ways run at odds with the vision of the future that they're trying to suggest for. Writes Maul, Sounds impressive and Meta can talk all they want about personal superintelligence. But as long as their core remains an ad-driven dopamine machine, they're the last company I'd trust with something like this. Matt Middlestad writes, Despite reporting that superintelligence labs would be startup-esque and walled from Meta's business as usual, these goals seem deeply tied down by Meta's social media legacy and existing product line.

22:38It's hard to imagine big transformations if we don't aim at big problems. So I want to talk about a few parts of this. First of all, let's acknowledge that this starts with a very profound argument, that Meta is starting to see self-improving AI. Now, maybe it would be too easy to overstate how far they're trying to say that is, considering he's using words like glimpses of our AI systems improving themselves, but he says the improvement is undeniable. So you have here another head of one of the major labs, arguing that we've passed some fundamental event horizon. Second, this piece is absolutely trying to draw a sharp line between a vision for superintelligence that's all about how you get your work done, and a vision for superintelligence that's all about your life.