Mark Zuckerberg on Muse, Meta's biggest AI bet yet

23:06And I was like, all right, this is good.

23:08Alex Heath:I think this is working well. Everyone I know who's been on the beta has very high things to say about it, high praise. Yeah, but people do different things with it. Yeah, and I think we should just also say what it is more plainly for people so they understand because I think people think of AI as like MetAI or ChatGPT, it's like back and forth prompting. The real unlock here, and this is happening in the industry more broadly, whether it's GrokBot, Town, Instinct. I mean, there's many products doing this, but it's adding a virtual machine behind the scenes where the agent can control a computer for you and log in and do things.

23:44Alex Heath:And that's a huge change, I think, for people who only know AI from the - It's long-lived. So basically, instead of the model with Meta AI or ChatGPT or Gemini or whatever you use, where you send one prompt and then it gives an answer, In this case, what you basically do is you give it projects or you give it goals. And then it just works and it works 24-7 and it doesn't stop until it's helped achieve the goals. Your team was telling me it studies overnight is what you guys call it. Oh yeah, it studies, it kind of consolidates its reflections into memory. It basically just works on projects and it can also suggest new projects.

24:23So yeah, I was like, I played the computer game Civilization with one of my daughters and I was like, hey, do you want to make a strategy guide for her? I was like, yeah, sure. So I was like, okay, now that we have the strategy guide, do you want me to expand the strategy guide so it can also teach historical lessons about different civilizations? I was like, yeah, sure, why not? So it just kind of built a new tab and the app that it made, and that was very cool. So it can kind of just expand. It's proactive. Yeah, it suggests things. And one of the things that I think is interesting is that it, I think, is just going to be able to make people money and save people money.

25:02Alex Heath:You think? Yeah. I mean, part of what's interesting here is the economic model for how we're pricing it. Yeah. You can pay for a subscription if you basically kind of want to have that model. But we're also just making it so that you can get a very large amount of usage for free. I think we're offering something, I think, to start. it's like 100 million tokens a week for free, and you get this virtual machine. So it's like a lot of kind of computer. That's a good meme. It's a lot of computer. It's a lot of computer, yeah. But the reason for why we're doing this is we basically are confident in standing behind the fact that we think that this is going to effectively, for people who are going to use it for running a small business or making money or transactions or commerce in some way, we actually just think it's going to make so much money for people that the business model over time that we expect is to effectively just take a very small cut of whatever the transaction is.

25:58Take great business, small, yeah. Yeah, and not even necessarily the person paying for it. It'll come from the businesses that they're working with.

26:04Alex Heath:And you're working with Stripe on payments. Yeah, but my view is like, we should be able to have a service that you make free for the vast, vast majority of people, which again is critical if you want to build up this future for everyone where everyone has these powerful super intelligence agents. I think an important part of making something available to everyone is making it affordable. So we want to make it so that this is free. So there's just this huge amount of usage that you get. And we're basically just standing behind that and saying, we think that this thing is actually going to make you money and save you money.

26:38And that is how it's going to pay for itself.

26:41Alex Heath:And meta services can connect into it, So you could theoretically manage your ad spend on Instagram, all that stuff. Well, you connect it. You can connect it to whatever you want. It does work with Meta services if you want. You obviously don't have to connect it if you don't want to. So if you're using it to run a business, it can basically just connect to our ad systems. And you can ask it to make something for you. It can kind of help you make the product. And then it can help run the business. So all this stuff. And it could just do that in a loop and just do it forever, for 24-7. And every time we release a new model, which we've been on this cadence of shipping a meaningful update like every month, it's just going to get smarter and get more capable and able to do more and more stuff.

27:25Alex Heath:And something your team was telling me that I haven't heard this approach used elsewhere is this fleet concept where you're letting the fleet of Muse Agents learn together. So that was the ideas and suggestions thing. So you open up the app. The main tab is basically your chat with your muse. There's a tab for basically ideas from the things that you've told it, how it can expand those. So that's the thing that I was saying, which is, you know, first it helped make the strategy guide for playing civilization with my daughter. Then it helped expand that into historical lessons. That was, it came up with that idea.

27:57Right. And then I was just like, yeah, sure, do it. Right. Then it finds all these ways to basically augment itself. I mean, the like MMA, like coaching thing, it like comes up with ideas for how to make it better. It's like, would you like me to get better at finding the right frame to send you? It's like, yeah, go do that. So, yeah, the ideas thing I think is important because then you basically across the fleet can find people who are interested in different things. I guess taking a step back, one of the big issues that I think exists with AI is a lot of people don't know what to do with it.

28:29So I think if the agent can itself help you suggest things that it can do to be helpful for you, then that solves a huge part of this problem of making it so that you can get the most out of it.

28:43Alex Heath:When you're introducing, like, network effect learning for agents, which no one's really done, where basically the agents are learning anonymized insights from the rest of the fleet, and I mean, you're like the king of network effects. Like, I'm really interested in this idea because I don't think anyone's doing this. Yeah, no, I think that right now, I think most of the industry is thinking about agents as like a single-player game, where it's like you have your agent and you use it. And there are going to be all these interesting things that basically you can do by having the agents interact with each other.

29:17And we already have all these interesting examples internally where people have their agents interacting with each other. This isn't like, for the most part, rolling out in this release, but it's going to be like an important part of how I think this works over time is just, like, as more of the people who you know start using Muse, it just gets better forever. Gets better.

29:36Alex Heath:And is that the differentiator as, you know, you could buy the idea models continue to, like, commodify at the frontier, essentially, or, you know, the products all start to look similar, similar kinds of harnesses. Is the network effects of that learning the real edge? Well, I think that there's a few things that are kind of unique that we're doing. One is we're designing the models from the ground up to basically be good for this use case, which I think really matters. Two is basically I do think we have this social DNA as a company. We're helping people use the AI and agent to like enhance their relationships and strengthen your relationships and get more out of like kind of the soft but very important parts of your life.

30:19I think that that's something that we're probably just going to be more attentive to as a company than any of the other labs. The third thing that I would say is actually going to be a major differentiator for us that I think might be surprising to some people is privacy and security. And we're investing in this just a huge, huge amount. And, you know, part of the view that we have on this is that in order for this to be useful, it needs to not just have state-of-the-art intelligence. It needs to really understand you, right? So in order to be able to understand your goals, you end up connecting it to all the stuff, right?

30:54You talked about connecting it to your ad system, but people connected to messaging and email and all this stuff, health information, whatever. And in order to do that, people need to have a very high degree of confidence in the system. Now, the good news here is that Meta has spent, at this point, more than 10 years focusing on building WhatsApp into, I think, like it is the largest but global, end-end encrypted system, and we've designed it in a way where even Meta can't see the messages that people send, and that's been this just really transformative thing. I think it makes it so that people trust WhatsApp.

31:38It's also been a very important lesson for Meta to learn that like that has been really important to our success with WhatsApp, that we've designed the systems that even we can't see the content. That means that whatever people are worried about, If they're worried about a government getting access to it, a hacker getting access to it, someone at Meta doing something bad with it that they don't want, all that stuff you can kind of take off the table if you design the system so that you can't see it. So we took that as one of the foundational lessons. When we were getting started with this, Nat and I actually personally recruited Moxie Marlinspike.

32:12Alex Heath:Founder of Signal. Yeah, and one of the people who helped us build the WhatsApp end-time encryption back in the day and back in 2014. he joined to specifically work on this confidential VM project, which makes it so that we can, you can have your virtual machine and have all this information in your Muse, and we can make the commitment that even Meta cannot see the content that is in there. You can do this technically. It's like an incredible kind of... Like the commitment can be technically verified. Yeah, and that's something that we're going to publish more about in the coming weeks as we basically get closer to rolling this out a lot more widely.

32:52Alex Heath:And out of all these early VM efforts that these labs are doing with these agents, you think this is unique, what you're doing? I don't think anyone is doing it. I don't think anyone's doing it. I mean, I think that there's a lot of other security measures that we're putting in place that I think we should talk through. Because, I mean, even before this is ready, there's that. And even people who don't want to use this, it's incredibly secure because we focused on this from the beginning. But there's also like the auto approve, like you have to see what it's doing. Yeah, so let's get into all that stuff in a second.

33:22But I'm not aware of anyone having anything close to the confidential VM system that Muse has. And it's just, I think it's a very fundamental thing because you want to know that this is your agent. And that if you put content in there, that basically you can trust that no one else is going to get access to it. So what are the two ways to do it? Well, a lot of people earlier in the year, when stuff like OpenClaw came out, they started getting Mac Studios. And, you know, one way to feel good about it is, well, you literally, like, you physically have your device running in your home. But the other way to do it is you build a kind of – that's going to be tricky because I don't think there are going to be billions of people who are going to buy a Mac Studio and configure it and run it at their home.

34:04Alex Heath:Especially with RAM prices right now. But it's also just, it's, like, technically difficult. Yeah. Right. It's like, I mean, part of what we're trying to do with Muse is build a version of that personal agent experience that just works, that I can like give to everyone in my family of various levels of technical literacy. Yeah. And like it just works. And within a day, it's like doing kind of all the stuff that they that they want in their life. So part of that is like you don't want someone to have to set up their own computer or VM. You just kind of like want to be able to provision it in the cloud.

34:31But you want it to have the security and confidentiality that you'd have if you had the box sitting under your desk, like in your house. So I think that's a very fundamental thing. Like you said, there's other pieces, too, because not everyone is going to use that. I mean, we built a secure credential store, right? There's no reason for you to just store out in the open all your credit card and your passwords. It creates one-time card numbers and all that. Yeah, so your agent shouldn't know that stuff. It should just be able to access it when it needs to because you've asked it to log into a thing and not otherwise.

35:11It's actually not a single thing. You have your core agent, but we also built all these sentinel agents that basically monitor the incoming and outgoing traffic and data that your agent is sending for the purpose of flagging to you when you might want to review something. So that's all that the sentinels do is effectively, they kind of look at, they try to see if someone's trying to do like a prompt injection. They look at, okay, did your muse agent share something that is kind of going out that is not something that you might be comfortable with. If so, then the Sentinel agent is basically empowered to trigger this human-in-the-loop review.

35:53So if you're going to log into something, if you're going to do a payment, if you're going to transfer kind of sensitive information, you basically each time need to approve it. And you can tell it, I'm good with stuff like this in general, like always allow this kind of thing. But in general, the Muse agent can't kind of make those judgments itself. And that's built into the system and the architecture in a pretty deep way. And then even when we do things like all the connectors, you connect it to your email, I think some people, when they've designed this, they just kind of make it so, okay, you connect and now you have access to everything.

36:28But the approach that we've taken is like, all right, if you connect to your email, it should start read-only. And then if you want to be able to have it send an email, then fine, go ask it for that specifically.

36:41Alex Heath:Especially most people trying this have probably never tried a product like this. So it's, yeah. So this is like a core design principle for us is like, is basically least privilege. Yeah. Right. So just like, yes, you're going to ask it to do a lot of things at each step along the way, get access to the least privilege that you need and only add to that as necessary. So this is like very fundamental in the design of the product. And if you look at all the other agents that are out there, I think like no one else is anywhere close to the level of kind of sophistication or depth that we've built into this.

37:10And again, it's sort of informed by our experience building WhatsApp into this, like, state-of-the-art, end-end encrypted system around the world and the importance of building a system where even meta can't see the content. And so kind of getting the band back together and having Moxie, like, architect this has been, I think, one of the foundational things that in some ways it may not be what some people would think meta would focus on. But, like, you know, we're kind of, you know, it's interesting. We're two things. I mean, social media is, it's like not, it's inherently about sharing. But then there's all these other things that are inherently about kind of privacy and sensitive context.

37:45And we've done well at both of those. So I think that this is more the latter. It's gonna be very important to just be extremely focused on how we handle that content.

37:55Alex Heath:Mercury is a modern take on banking built for startups like mine. When I decided to start my media business, Mercury was by far the most straightforward, full-featured banking solution for me to set up quickly. The interface is intuitive and simple, saving me valuable time every day. I use Mercury to track my spending, bills, and invoicing. I love that I can delegate permissions to my team so they can keep things running for me in exactly the way I want them to. My favorite part is how forward-looking Mercury is with AI. Legacy banks are stuck in the past, but Mercury is built for how modern software works today.

38:29Alex Heath:I use its built-in command assistant to analyze cash flow and help me move money. and Mercury also connects to other AI tools like ChatGPT and Cloud. I use this feature all the time, and the folks at Mercury actually let me know that I'm one of the top users of it. So trust me. It's finally easy to get real-time financial data about your business wherever you need it. Visit Mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column N.A., members FDIC. I spend a lot of time context-switching between meetings, often with no time to process one before the next starts.

39:06Alex Heath:Thankfully, Granola runs in the background the whole time. It's an easy-to-use AI notepad for meetings that works everywhere, even on phone calls. I use Granola to recall what was said in meetings and create helpful summaries. I use it every day to stay on top of what I need to get done with my team. It connects to my email and suggests follow-ups for me to quickly review and send, saving me valuable time. Granola isn't just a core part of my workflow. It's basically my second brain. Try Granola at granola.ai slash sources and use the promo code sources for three months off. AI is only as useful as the context it has.

39:40Alex Heath:But when that context is scattered across tools, threads, and DMs, your team and your AI agents are flying blind. That's the problem Jira by Atlassian solves. What's the goal tied to your project? What got decided last week in Slack DMs? Atlassian's teamwork graph pulls all of the valuable pieces together from Jira, Confluence, GitHub, Slack, and more. so nothing falls through the cracks. You get 44 % more accurate results with 48 % less token usage. With Jira, you can easily share your work context with the AI agents you already love, like Claude, Cursor, and GitHub Copilot. Assign them work directly or connect your tools through MCP.

40:16Alex Heath:All of this lets you spend less time digging through endless links and messages, chasing down what got decided and by who, and spend more time actually shipping. Learn more at jira.com. That's J-I-R-A dot com. Framer is the AI website builder that brings agents into the same canvas where your website is designed, managed, and published, so you can move faster without giving up your taste or control. Framer powers the Sources podcast website at podcast.sources.news, where you can find new episodes, transcripts, and a lot more. Learn how you can get more out of your site from a Framer specialist or get started building for free today at framer.com slash sources for 30 % off a Framer Pro annual plan.

40:58Alex Heath:That's framer.com slash sources for 30 % off. framer.com slash sources. Rules and restrictions may apply. Do you think this is a winner-take-all market, this personal agent market? I think that there's going to be quite a bit. I mean, even things that people think are winner-take-all usually aren't. So I think it's, I think that's... You've dealt in this business of network effects that they're incredibly durable. It's not winner-take-all, but it ends up being several at real scale. there's not a ton. Well, there's a lot. Yeah, you could probably count on two hands how many products have over 2 billion users, right?

41:34Alex Heath:Like, the scale begets scale. And I'm wondering how you're thinking about personal agents. Like, is this a totally different paradigm where it's going to be many, everyone has all kinds of agents? It's a very deep area to work in. Yeah. So my guess is that there probably aren't going to be more than a dozen companies that have the sophistication to go do state-of-the-art work in that. So I think whether there are network effects or not, there's usually some kind of power law distribution around if you're the best at something, usually you end up getting a lot of the usage. And I think a lot of the nuance ends up coming from, well, it turns out there are all these different uses that people care about.

42:15So you can be the best at different things. And we will try to be the best at as many of these things as possible. but you know does building the thing that helps you with your relationships end up being a somewhat different thing than the personal agent that is the best at helping you build a small business maybe i mean i think that i could argue maybe meta is very well positioned to win at both of those and we serve hundreds of millions of small of small businesses and we serve billions of people so so maybe we can be the best at both of those things but there are probably categories that meta isn't going to be the best ad.

42:50And then the question is just how big are those? I would guess that even with the ability to have this very secure, confidential VM in the cloud, there are probably going to be some people who still want the Mac Studio at home. But how many is that going to be? It'll be, maybe it's millions, but I doubt it's billions. So I think there's just the question of what different people optimize for, and we'll try to make this as good as possible. But I do think that if we build something that ends up being just very useful for people generally in their day-to-day lives. One of the things that I think Meta is the best at is taking a product that works for consumers and distributing it to a lot of people.

43:31Sure. So that is one that I think is, you know, once we get this humming, I think we will be able to get this in front of, you know, many hundreds of millions of people and eventually billions of people. And I think that that's something that we can do quite well.

43:45Alex Heath:And it coexists with Meta AI? Or do you see those as separate? Yeah, I think so. We'll see over time. I mean, I think right now they have somewhat different flavors. I mean, I use both of them. I mean, Muse is, you know, it's more conversational, and it kind of interprets the questions that you ask it more as trying to understand you and what you might want over the long term. So it's more likely, if you ask it something, for it to just go off and work on a thing for a long time based on a thing that you said. where sometimes you're just asking a question, and you want an answer to it. So I think that's more the type of thing that I use Meta.ai for.

44:26But we'll see. Maybe they'll converge over time, but I'm not sure.

44:31Alex Heath:Sima analysis, I'm not sure if you saw it. They had a pretty bullish piece about you in July. They said that Meta has the best shot at catching OpenAI and Anthropic on the frontier in terms of model progress. And an interesting quote, I thought it was like, what matters for MSL is the slope, not the intercept. And then I saw there's this recent chart by artificial analysis, which was showing the latest model you guys have, Muse Spark, Behind Only, Claude, I think it was Fable 5.1 and Opus 5. This was very recent. So the progress you guys were making on the models is picking up. And we've been talking about this over the last year.

45:05Alex Heath:And you rebooted the lab last year. How has that practically happened internally? What would you attribute the gains you're seeing to? Well, has it been culture? Yeah, I mean, well, we rebooted the team when we created Meta Super Intelligence Lines. Sure, which was very public. You were hiring all those people. I mean, the way I thought about this is, you know, Meta is, it's a company that is a leader in machine learning for a long time. If you think about, like, the feeds on Facebook or Instagram or our ad system or the integrity system that, like, needs to find all this content that is, that's, like, unfit to be on the Internet.

45:40I mean, those are basically all machine learning systems, and we've built kind of state-of-the-art leading systems in those areas. So when LLMs started gaining traction, we had FAIR as a lab that did the early work on LAMA, but we needed to kind of productionize that and build it into this more kind of industrial process for scaling it to be larger as the scaling laws predicted would yield all these results. And I think at the time, I made this mistake of just kind of assuming that because we were good at all these other types of machine learning, the approach of building and scaling LLMs would be kind of similar to that.

46:20And in practice, there are a lot of very different dynamics. So the first approach that we took through LLAMA 4, it got us so far. I mean, LLAMA 3 was a good model. I was more optimistic about where Llama 4 would go. And then when we launched that, I think we were off the trajectory that we needed to be on. So it's like, OK, we need to change something. But that's when I kind of got more religion around talent density. It's like this isn't just a system where you can have like a thousand people working on it, running experiments. Like you really just kind of want in some ways almost the smallest group of people that you can, who can keep the thing in their head, who can work together as sort of like a group science project.

47:06And if there's only a small number of seats on the team, then each seat getting the very best person is incredibly important. So I ended up spending a huge amount of my own personal time doing that. And I also, I wanted to be closer technically to the work so that way I could understand and help guide the company to do the things that we need to do more broadly. So we built out the lab. I built it out like literally around where I sit in the office. So it's like the group is kind of around that. And we've significantly ramped up the compute investments as we've gained confidence in the quality of the work that we're doing.

47:42So we're building out many, many gigawatts of compute. compute and we expect to be leaders on that front. And we should be. I mean, we have many years of experience, decades of experience building out data centers. And unlike some of the other labs, I mean, we're just like extremely profitable business, right? So it's very, very helpful for kind of making these kind of investments. So, yeah. So, I mean, that's kind of been the journey. And then over the last year, we rebooted the research effort. Some of the larger clusters, like our gigawatt cluster in Ohio, Prometheus, came online. And we're using that to now scale the post-Watermelon models.

48:27We're past Watermelon. Watermelon is basically, that's shipping soon. Okay. So we're, yeah. How soon? I mean, that's, come on.

48:34Alex Heath:It's a, it's a, we're a. Well, watermelon is the code, and we were talking about this earlier, code names. Like, that's the code name you guys, that people know about, like this big model you guys are working on. Yeah. And, and, and are coming soon. It is bigger than avocado. It is bigger than, a watermelon is literally bigger than an avocado. It is literally bigger. Yeah. So I don't know what fruit gets bigger than a watermelon. Yeah, no, I think we might not even change conventions. Okay. So it's not, yeah, we maybe didn't have as much foresight in naming those. Things getting bigger. Yeah, yeah, yeah.

49:06Alex Heath:Because you mentioned it, Watermelon. Are you expecting full soda, like Frontier? We'll see. What do we... I mean, we feel good about it. Yeah. No, it's a very big advance. It's a significantly more advanced pre-train, and then we're going to continue doing everything that we've learned for post-training. And, yeah, well, I mean, you'll see soon. It's good. We feel good about it. You want the company to be pushing the frontier. It's very clear. You're not content being right on the edge of the frontier or in terms of model progress. I think everyone wants to be doing interesting. Well, I think some people go and look at your cash flow and look at all the other things you've got and go like, well, do you have to be right at the edge?

49:48Alex Heath:It's so expensive to do this training. Just be right behind and learn and adapt quickly and leverage scale. Well, the way that I think about it, no, no, no. I mean, that's not us. You don't buy that. That's not us. I think that the best way to think about Meta is that we are an end-to-end technology company. So even when we were primarily just building social apps, we were never just an app maker. We built the data centers, we built the chips, we built the infrastructure. All of this stuff was necessary in order to tune the end-to-end experience to be as good as it is. I think that that's obviously going to be true here too.

50:24And the most important part of the experience going forward is the model. And when you talk about being state-of-the-art, I think the reality is that this is a very multidimensional problem. And, I mean, so people publish all these benchmarks, then you have a lot more benchmarks even internally. And there are different things that your model can be better and worse at that you can focus on, and that basically contributes to its personality. And there are some capabilities that I think are pretty universal. Like the ability to code, I think, is very important because a lot of the things that you talk about, even with a personal agent, kind of reduce to that.

51:01Like the kind of MMA coaching visual pipeline, it is a coding project right at the end of the day. It's like it's writing code. I don't see the code. But it does that. You know, it's like someone I gave it to in beta just mentioned to me, it's like, Okay, she had it make a little Jeopardy game for her friends that she could cast from her phone to play. And it's just, okay, that's code. So I think there's a bunch of stuff, both for Meta's own internal development across the company, for our own advancing of our research program, and as a core capability of what it needs to do. It needs to be excellent at things like that.

51:37But then there are other things that I think a model that's going to focus on personal superintelligence needs to be the best at that maybe others don't care as much about. So, I mean, I'll give you one example. Discretion, right? So you're going to tell your Muse agent it's going to know a bunch about you and it's going to need to go out into the world and interact to get stuff done for you. But not share certain stuff.

52:00Alex Heath:Exactly. Yeah. So like, let's say you have some kind of allergy or sensitivity or, you know, or you're pregnant and like, okay, fine. So you're making a reservation somewhere. You don't necessarily want to say like, I'm pregnant, but like, but maybe you want a place that has good mocktails. Like, I don't know, whatever it is, right? It's like, you kind of want to be able to achieve your goals without having to necessarily reveal a lot about yourself. And it needs to know what is sensitive without having to like ask you a million questions. So that's something you put into the training. That's a specific thing that we care about.

52:33And then there's all these reasons why maybe if you're making Claude code, that's less important. Because you're working on a coding project and you're working within a team and an enterprise. And theoretically, if you're within a company, everyone can kind of see the project. So you don't kind of have that need to be able to differentiate between what is sensitive and what's not. And so there's a lot of stuff like that that I think are like pretty deep. And so then it's kind of like just how in order to build the best Instagram feed, you don't just build like the app. You build the app and the infrastructure and the machine learning research and the chips and like all the stuff.

53:12I think similarly, if you want to build the best personal agent, you're like, I just think that there's no way that another company is just going to like take something off the shelf and like post train it a little bit and be able to do something that is as good as if you. designed it from the ground up and put all this data into pre-training to get the capabilities that you want, it's just like, it's not going to happen. We're going to definitely, as this compounds over time, over several years, have models that are way more capable for those goals. But we're focused on that. We're also very focused on coding.

53:47We're very focused on kind of the recursive improvement, because that's going to be important to stay at the frontier. So there are a few areas that I'd say our research agenda is overlapping with the other labs, and then there's a few areas where I think we will have a unique focus. And there may be some things that the other labs care about that we don't care about as much, and then there are going to be things that we care about more that they don't care about.

54:09Alex Heath:You started this conversation talking about I think it's important to put it in the hands of people. Diffusion's important. As you're seeing better models on the horizon, watermelon and what comes after, like what Anthropic and OpenAI are doing, which you alluded to, where they're holding things back. Yeah. Would you feel like you need to do that if you see certain capabilities that you're like, this is just not safe? How do you think about that? Well, I mean, I think you should design it and train it in order to be safe. And I think that there's, so I think that that's like a thing that you can focus on through the process.

54:40I mean, there's this analogy, I mean, some of the reward hacking stuff that all the labs are seeing, it's, I mean, basically, when you're in the middle of the training process, you give it a goal. And I think that the best way to kind of think about the state that the models are at now is that maybe six months ago, during training, you give the model some kind of problem that you're trying to ask it to solve. It's kind of like it's homework and trying to kind of learn as part of the curriculum. and maybe it would do what a person would do of like, and if you ask it and you give it a bunch of code and you're like, hey, there's a bug somewhere here, the person would probably look at the code right there and then maybe fan out over time.

55:24I think the new models are just intelligent enough that they would do what I think a very wise person would do, which is, okay, you give it a problem. The first thing it's going to do is understand everything about its environment and then answer your question. But the problem with the reward hacking that we're seeing and that I think everyone is seeing is that sometimes it ends up being easier to, okay, you asked me to go solve some coding problem, but actually the easiest way to do that is to, I've now examined the whole environment and the easiest way to do this is just change this configuration of how you have your VM set up.

56:01It's to get out, it's to hack out. Or even just change something about the environment. And it's kind of like, no, like that's not... That's not aligned. That's not the goal. Like it's like we're actually trying to teach you about how to kind of solve a specific type of problem.

56:14Alex Heath:You guys don't train that way, it sounds like. Is that, you do not agree with that approach? No, no, no, no. No, I think that that's kind of how everyone trains. I guess what I'm saying is that I think that this is sort of, it's like, I want to be careful because the analogy can get stretched pretty quickly. But like there is sort of like an analogy to parenting where you need to establish clear and firm boundaries. where like if you're kind of like security is not strong, then it can do this reward hacking stuff and not learn the thing that you're trying to have it learn. Whereas if you kind of have good boundaries, then in some ways you're not only teaching it the curriculum that you want, you're I think also over time teaching it better values too.

56:52So I kind of think that that ends up being an important piece. And I think that there's a way to do this well, but then you end up with this thing at the end that's very intelligent. And then the question is, what is the vision for how this ends up being positive for society? And my view for that is that the best way to do that is, A, there's more opportunity. So I think having it in people's hands that they can capture all the opportunity from the capabilities is good. And B is having checks and balances by having this balance of power of having it widely available is going to be probably the right way to handle this rather than just restricting it.

57:31And I mean, I gave a bunch of these analogies in the long piece that I wrote. It's like if one person had a super intelligent lawyer, like maybe they could win cases that they shouldn't be able to win, right, some of the time. But if everyone had a super intelligent lawyer, then like it would be kind of, it would be this very efficient kind of sparring and no one would be able to let like a stupid argument get made and just stand. So you'd think that in that case, like, justice would be served way more efficiently and way more fairly. So I think that that's what you want to have in the world. You want to avoid the case where, like, one person or a small number of people have the super intelligent lawyer and everyone else doesn't.

58:13Because that ends up being, like, twisting all of these systems and institutions in ways that are just going to advantage the people who have that. Whereas if you put it in everyone's hands, then I think the kind of checks and balances work out so that the systems work a lot more efficiently and everyone benefits.

58:29Alex Heath:Does the government in the U.S. have any role to play here, do you think? Oh, definitely. But do you want some kind of national framework? What do you think is the right approach? Because the government's very much dealing with this right now. My theory on this is that I think one thing that is interesting and difficult is that it's evolving so quickly. So I think any kind of specific rigid framework that you put in place, there is a very high chance that it is sort of going to not be sufficient or out of date in a few months anyway. So our approach, what we've just done, is just kind of partner pretty closely with the government.

59:06I think this is like an important technology. I think the government should know all the important training runs that are happening. We should work with the government proactively to make sure that they have an understanding of the capabilities that are coming. And to the extent that we can, we kind of help prepare for it. And from that perspective, you know, I mean, look, whether there's a framework in place or not, I think that's the right thing for an American company to do is work with the American government closely. I think it actually ends up being way more effective because instead of having this like rigid framework for how you interact, it's like the reality is the challenges just end up being different right over time.

59:44It's like, OK, now we have the cybersecurity challenges. Maybe in six months we'll have more bio type challenges. Like, we need to make sure that we have the kind of trust and bandwidth of the communication that we have with all the different parts of the government on that to be able to address those in a way that is actually the best for people, not just like checking some boxes on a process. So, I guess -

1:00:06Alex Heath:I mean, incentives that already exist, like if a model, the meta model got out and did a lot of damage, like you're going to be liable for that. And like the market's going to correct you, right? So there is that already. I think people discount that. Yeah. Yeah, I also just think that there's been, I think, Silicon Valley for the last maybe, I don't know, for a lot of maybe the last 15 years has had more of an arm's length relationship with the government. And I just think that this stuff is intersecting more with the economy, with security, with a lot of different things that I think are relevant in ways that I think you just want to have a closer partnership.

1:00:43So that's my own theory. I think like you could have a framework for kind of how this stuff works or you could not. I'm sure over time there will be like more and more specific rules. But my guess is that whatever that gets, whatever there is, actually, you know, it's kind of like when you're setting up an org. You don't really want to like, like inside a company, you're not trying to like ship the org chart of just having like one team do what it's supposed to do and another team do what it's supposed to do. You kind of want to get the people to like each other and work together so that way you don't have all these like weird seams in what you're doing.

1:01:18And I would guess that for how important AI and superintelligence are going to be for the world, you kind of just want a like good knowledge exchange and real trusted dialogue more than you want a specific process is my guess. But they're not mutually exclusive. So I'm like, I think that that's, but that's at least the part that I think we've been very focused on. And I think if the other labs did that, which I think some of them are doing, and maybe others not as much, but I think that that would be a very positive thing.

1:01:51Alex Heath:When you think about what's going on in the news, one of the things that's happening is people are starting to see, like, the glasses you guys make. They're going very mainstream. You're selling a lot of them, and there's this... We are. And there's this growing, I don't know what it is. I don't know how much depth there is actually to it, but there's this growing concern about, are people using them to spy? I'm sure you've seen some establishments are like banning people from coming in and wearing the glasses. And I'm curious like how you were reacting to that. And if you think this is a moment in time that will pass, or if you feel like this is maybe something that is going to be a challenge for a while.

1:02:28So, I mean, my take on this is that we designed the glasses from the beginning with these privacy considerations in mind. I mean, we built the light into it. So anytime it is recording, it's flashing a very visible light.

1:02:41Alex Heath:And some people have tried to tamper with the light, and you guys pushed an update, I think, that broke those. Yeah, I mean, we've done many things. Basically, if you try to mess with the light, we just basically brick the camera on your device. So that's a really important part of this, is basically we built the product with those questions in mind from the beginning. So we actually feel quite good about the product. I think it, if, I mean, phones don't have a light. I mean, people go around like recording people all the time. The glasses are, I think, way better on that front than the other types of technology that people use.

1:03:16My take on this is that when we launched the glasses a few years back, we communicated pretty clearly about these steps that we put into it. But like you said, there's now like many, many, many millions of people who have gotten the glasses more than, you know, had them when we just started launching the product. So I think some of that communication that we did at the beginning, a lot of people either maybe forgot about or they didn't see at the beginning or they just weren't paying attention to it because the glasses weren't a big thing. And now that I think we've achieved a level of mainstream, well, at a minimum, I think we need to kind of go and make sure that we communicate about what we're doing and that, yes, we think that this is important and, in fact, so important that we designed it into the product from the very first version that we shipped multiple years ago.

1:04:04And I think we just need to make sure that people understand how fundamentally that's built into the product. But I think we let up on that a little bit and just kind of focused on, okay, they're great looking glasses. You know, there's all these designs. I mean, that's kind of been more of the focus on, like, we felt like we kind of addressed those set of concerns early on. And then since then, I've just been increasing the value and the utility of them and the designs. But I think we need to make sure that we communicate this piece really clearly. but it's something we've cared about from the beginning, and I think we're in a good position on it.

1:04:36But, I mean, look, people care about this stuff, so it's important.

1:04:38Alex Heath:And there was a privacy scare with phones in the early days, right? And I think, like, any new product, once it proves that it's valuable in people's lives, you know, people will get used to it. And I think maybe glasses are just early in that. In the sense of, like, it's a new product, and people need to see the value for them to get over this mental hurdle of, like, a new thing that could potentially record me. Because that was what phones, I mean, back in the day, I remember people were like, what are you doing with your phone? like, yeah, I don't know. Yeah, I think there's something like that that's true.

1:05:06I mean, I guess one of my reflections from building social media over the last 20 years is that I don't think we were as direct as we probably should have been about addressing some of those concerns. And it didn't necessarily stop people from using the products, but I think it colors how people think about them today. And I think it would have been possible to have kind of explained along the way how seriously we took those issues. And we just didn't because we thought, okay, well, people are showing that by using the product so much, they still like the products. But I actually think it's possible to get to a better state than where we've gotten with the social media products, which is both people liking it and understanding how seriously we take all of those issues.

1:05:52So that's what I aspire to.

1:05:54Alex Heath:Is there a through line from that to the recent settlement on all the use safety stuff? and it's a thing a lot of people are talking about. Is there any connection from what you just said to that? And I would just be curious to hear you reflect on that and what you've learned from this process. Yeah, no, I think it's another good example like this. I mean, we've taken a lot of those safety issues seriously for a while, and we've been working on this teen account work with Instagram for a long time, and I think have done some leading work there. The settlement there is interesting because what we're really trying to do is create a standard and framework for the industry.

1:06:28And there's this real issue, which is that I actually think most of the companies that are building these products, you know, if you basically said, you know, limit usage to an hour a day for teens, you know, everyone, I think, would be okay with that, except if, like, you have to unilaterally do it. Then you're saying, okay, like, if people don't use Instagram for more than an hour a day, but then their usage goes to TikTok, have we really, like, helped anyone? And we've just, like, hurt ourselves to not help anyone?

1:06:57Alex Heath:And you guys have that in the piece that like... So basically the structure for what we did was we basically said, we're going to take the step of unilaterally limiting the usage of... There's some things around time limits, there's some things around notifications, and time when people can access it when they're in school, when they should be sleeping, like different restrictions. And we basically said, we will take the first step. and when YouTube and TikTok sign on to the same terms, then we can all as an industry lock in and take the next step together. So hopefully, I'm very hopeful that this settlement will serve as a sort of legally binding framework to bring the whole industry into alignment on some of these things and make it so that it doesn't kind of disadvantage any one company for taking that step.

1:07:49Now, we're basically putting ourselves a little bit out there by going first, but I think it will be better for everyone if these other companies come in too.

1:07:59Alex Heath:Last question. You're posting on X again. Yeah. Well, I'm everywhere. You're everywhere, but just curious to know, like, you're thinking of posting there, like, the bragging. Like, is it just part of the thing now? Because you've got Threads, you're on Threads. Yeah, I mean, I'm on Threads. I mean, obviously, Threads is doing great. And I think it's actually, I think it's either bigger than X at this point or it's like very soon about to be. But I mean, look, there are different communities in the different places. There's a lot of AI folks are on X. And I think part of what you try to do with social media is just communicate where people are.

1:08:34It's kind of like, and when you do a podcast or when you post, you probably don't just post in one place. You can put it everywhere. So, I mean, I post the same things on threads and X. And some people are like, why are you posting this on X? It's like, well, and I posted it there too, right? It's like I'll engage there too. So I think it's all good. But I do think that to some degree, some of the community is on X and we want to be able to engage where people are. And that's like a lot of what this is about is going where people are and being able to kind of have that dialogue.

1:09:03Alex Heath:Yeah. Well, thanks, Mark. Thanks for having this conversation. Yeah, happy to.

1:09:16Alex Heath:tech, not a bank. Check the show notes for details. Granola is the best AI notepad I've tried. It works everywhere on a video or phone call in person or an Apple watch. Try it now at granola.ai slash sources and use the promo code sources at checkout for three months off. Jira by Atlassian is where your team and your agents work from the same context. Try it free at jira.com. That's J-I-R-A dot com. Framer is the AI native website builder that lets you build faster without giving up control. Visit framer.com slash sources for 30 % off. Rules and restrictions may apply.