Anthropic is training to destroy your company | EP 27

29:09The other options you have in terms of a harness, OpenClaw was the first, way too hard to install. You're going to be in system prompts. You're not going to know what it's doing. You're going to be upgrading it constantly. then you look at grokbot it feels like um a an app that was built by apple like weather or calendar or mail where there are no features in it it's just like you want the weather here's the temperature and then here's the next 10 days and we're not going to go too crazy right versus you know weather.com's app right where they try to go crazy or the stock app in on an iphone is the most basic, but if you want something more robust, you would go to Robinhood or E-Trade or something else.

29:55So that's what you should think about in terms of how it's designed. Now, how it works, when you're building in perplexity computer, which I love, when you're building in Claude Cowork, you have like two sidebars. You got the one on the left with all your projects and all your recent searches. You got the one on the right that's giving you all the usage and the skills and the data that's been put in there and, you know, your memory and what skills you created. All of that subtracted away and they just decided you're going to talk to a bot and it's like jarvis it's like how 9 000 or whatever that was called it's like samantha it's very similar in that it just has a conversation with you now you can do a slash and it will give you you know slash google drive slash x right but it is very powerful in understanding what you want and i create it You asked me what I've done with it.

30:49I'm going to pull it up here, and I'm going to go right through it. And I'll just tell you project by project what I did. So you and I were having a discussion today about systems thinking, right? We were talking about how to build systems, and this has been something I've been talking about. Hey, there's people who don't understand basic systems thinking, and then there's some people who they think in systems already. Creative people, people who make art, they tend to think with taste. They have great taste, but they don't do systems things. So I was like, hey, make me a playbook and find me all the resources on this.

31:22And then I told it what the circumstance was. And it came back and it made me a 12 page PDF to share with you and the rest of the team on how we could have a better playbook and all the books you could read about systems, seeing all the YouTube. Okay, boom. That's very similar to any other thing. Right. But I wanted to do competitive intelligence. and I took one of the verticals we're working in and I said, these are the titles of the people at the competitors we compete against. Here are 10 example brands of brands we compete against. Here are five titles. I want you to find all these people on LinkedIn, on X, on Blue Sky, et cetera.

32:03Make me a database, all of them. And then every day, give me a report on what they talked about in the last 24 hours. And if there's any business lessons, that I should know about that. And this competitive intelligence thing is operating every day for three days for me. And I feel like it gave me five or six incredibly actionable ideas that I would never have come up with. And it put them into Google Sheets. And it just kind of knows the next thing, the next thing. And because I can, on my phone, when I leave and then I'm on self-driving and I have an idea, I just put on Whisperflow, I talk to it, bang, bang.

32:39It just iterates on it. I did another one for hiring. And then I did another one for trends in AI for this very show saying, hey, these are the people who are interesting for trends. I run a report through Claude every week for this show. And it's getting better, but it needs a lot of feedback. You know, like it still gives me some outdated stories. It gives me some stuff that's like, well, we wouldn't talk about that. So, yeah, you need to walk it through. I want to say this is, all the, Claude has got too many features, and I think all the feature set and the difficulty it is to work with OpenClaw, way too difficult.

33:22Hermes, way too expert mode. These are for the top 10 % of the audience. Yeah. Because this is for the other 90%, and it happens to have a fidelity that I think is really strong, that combination is going to be the iPhone moment. it's an iphone grokbot is an iphone moment all of us had kindles um and um palm pilots and nokia n95s and flip phones and docomo phones that you know the nerds had sidekicks whatever blackberries then the iphone came and everybody else in your life had this this is what zuckerberg wants to build but it's here today it's well worth checking out it feels like something my mom could be building on and she's tech savvy but you know she doesn't do it for a living and so yeah I was blown away I am blown away don't think it's going to replace the other things for me but it might Eric are you on any of these big agentic platforms your Hermes or your Grokbots or whatever well I just downloaded Grokbot and I think what sets it like this newer wave of agentic systems what sets them apart is and like the new design pattern is that you want them running for days on end.

34:38And what that requires is, you know, a persistent computer. And you also want them acting on your behalf. And so you need like persistent cloud sessions that kind of follow you around every single device that you have. And so I think that those design choices made a ton of sense and make it like really, really useful. So I'm excited to use it a lot more. This is actually how we architected our platform, which is Silico, which is our interpretability agent. Interpretability is a research task that requires, you know, often like minimum 12 hour agent runs and agentic traces. And so it needs its own persistent computer, its own persistent GPUs in order to run these jobs.

35:22And so you architect it in a way where you can just kind of ask it a question and then you leave. And then, And three days later, it solves your problem and it reports back with all of your things solved. And so I think really this is kind of the future of any type of agent that you interact with. It should kind of just kind of live in the cloud and have a persistent session that you can pick up anywhere that you've left it off. Can we just double click on Silica? It's in my notes. I wanted to talk about it. It just launched in August. So it's an autonomous agent that plans and runs long horizon training experiments.

36:01Can you walk us through like what – give us an example of what might one of those long-term training experiments might be that you'd use something as powerful as Silico for? Yeah, it just launched in August. I'm really excited about it. It helps anybody run interpretability experiments and other training experiments on any kind of model that they want. So it can be on your own model. It can be off of any model that you can download off of Hugging Face. such as KimmyK3. And you can ask it a question such as, why is my model reward hacking in these scenarios? And then it can go in and analyze the model's internal representations, go and try to isolate the reward hacking neurons and circuits that the model is using in order to reward hack, and then also kind of surface the agentic traces that show when it's actually a reward hacked.

36:50And you can analyze deep questions about your model. It is a niche product. It's for AI researchers. And so specifically, like these are folks who are digging into the guts of these models every single day. But yeah, I mean, in order to run these experiments, these are long running experiments. And I think increasingly, not just with silico, like this is what GrokBot is clearly designed to do. It's like you want your assistants to be able to maintain persistence and massive long running sessions. And I think this is just kind of the future of pretty much everything that we interact with on a day-to-day basis.

37:24Yeah. And I think, Eric, it helps a lot of kind of like the industry that is focused on the applied aspect of things, right? I think as open models are getting better and better, you know, a lot of us in the industry are building their own models, right? At the end of the day, Jason, as you said, you know, you don't want to be dependent on one model, right? Like different customers have different preferences. You need to basically use it as a computational resource. And I think being able to explain the benchmarks and the results, especially for something like security versus just, you know, here's the output.

38:04One model is better than the other. That becomes super cool. So I'm glad we met on the podcast. This week in AI, we bring people together. Yeah. You guys are in a chat room together. That's one of the benefits of coming out of the pod is we put people into chat rooms together so they can build that fabric and we build that. You know, the thing I think we should just hop to is this breaking news story today, Lon, of AirPods with cameras. Now, I had somebody on This Week in Startups talking about AirPods with cameras. They were building something like that. Well, something leaked inside of the next operating system for Mac, and it's a guy holding up a book.

38:46and then saying like, hey, put this on my wish list or something to that effect. This has been a rumor that's been around for a while, but the idea is the AirPods have some sort of a camera in it. The camera is encrypted and not allowed to take pictures or videos. In other words, it's crippled. Its only function is to take a snapshot of something like on demand, I guess. And so Apple is now being faced, quite literally faced, no pun intended, with a fork in the road. Everybody wants to record the real world and then put that into memory. That's kind of the opposite of how they've built every product, where if you want to record something with your iPhone, it puts that like red bar or something on the top level.

39:34They're just very thoughtful about privacy, right? They don't want you taking snaps or videos or audio when you shouldn't be. but the trains left the station. But I thought this was like a very interesting possibility. And when Zuckerberg and Snap and Google and other people released this, Microsoft, I feel like this is going to be required for workers. And I just want to put that out there for a second of, imagine a world, in a world, where every worker is required to wear AirPods with cameras or smart glasses and record everything they do to program the organization's models. Yeah. Super God?

40:14Jarvis? Right. But it's also, we were talking about that, like, recreating employees who've left after they're gone. The ghosts. Ghost employees. If you record enough of someone's workflow day after day, you can then simulate them even if they quit. That's really terrifying, I think, in some ways. No, no. You got it all wrong. It's not when they quit. But it's when you fire them. Right. When they've trained themselves to obsolescence. And it says, you know what? Nothing left to learn from this sack of organic material. Right. Please bring me another sack of organic material. At that point. If the company can do everything you can do without you, there goes your only benefit.

40:57Galena? Yeah. I mean, if that's what it's going to take for us to go on vacation in August, then I'm all for it. But, you know, if it's got an upside, right, there's an upside to that, right? And to having only one company in the world. But I think if any technology has the capability to do anything, it can also be hacked to do something else. I mean, even if you're not intending it to to have it that way. So I think privacy, the way the world is going is effectively dead. Or we would have to fight very, very, very strongly to like contain it. But as Jason said, I think the train has left the station and it's crazy.

41:46Eric, any thoughts on the persistent recording of organic objects in enterprise organizations in order for the great result of score in AI reward learning? I mean, I think people really clowned on meta when they kind of drafted all of these meta engineers to go and just create RL environments every single day. And it turns out that maybe that was the right decision to help them catch up in the AI race. And we also just, Google just put a price on, you know, what, 34 years of airline data, buying all of Spirit Airlines data at auction. uh and yeah i mean that everyone is just starved for data and will like do anything it takes to get there and so yeah i guess like the train has left the station in terms of privacy and uh and yeah maybe humans are just like increasingly being used to uh yeah just like train train ai models and that's there yeah i don't know i it's this is like worth double clicking on eric i'm so glad you reminded me of this.

42:55Google bought Spirit Airlines, which is now defunct, their entire data set. So they bought the assets of the company, which when you go into bankruptcy, all the privacy concerns and hand-wringing, all that gets washed out by the judge. The judge just says, what can I get for this asset? Whether it's a customer list or behavior, whatever, and they just sell it for the benefit of the creditors. So what happened was there were hundreds of emails, phone records, everything these teams were working on and they held an auction and i think the next bid was like 7.5 million but anyway google bought 100 million emails 500 million items for microsoft teams 17 million one drive files 20.5 million items from sharepoint search giant also owns over 30 million recorded customer service calls recorded customer service calls and more than 15 million customer service chat records, 600 ,000 ServiceNow tickets, and for the low, low price of$10 million.

43:53And if you think about what you would have to do to recreate something like that, it would be hundreds of millions of dollars. Run an airline, literally run an airline. Yeah. Well, or if you were Micro One, which we're lucky to be investors in has done really well, or any of their contemporaries, you would hire a couple of hundred experts. You would ask them questions about running an airline, running customer support, whatever it is. The problem with this is there's no human in the loop, Eric. There's no human to tell them why they made certain decisions exactly. They have to infer it. Totally.

44:28How much value is this? Is it greater than$10 million in value in the future, or is it less than$10 million in value in the future, Eric? Yeah, well, my tongue-in-cheek response here is that they paid$10 million for 34 years of data on how not to run an airline and do a terrible job of doing this. And so this is why, you know, this is why, you know, alignment is important in that, like, yeah, you're training all of your models on terrible data every single day. And you want to make sure that you're learning the right things from your data, not the wrong things. And so this is our whole vision and mission at Goodfire.

45:08It's how do we actually select and choose what you actually want to learn from training data? You want to pick the good parts out and you want to reject the bad. You want to help the model be great at cyber without autonomously hacking hugging face and you reject those gradient updates from the model. And this is just maybe a little bit of the how the sausage is made in the ai industry where all of the data is like kind of bad uh and then you still get these like enormously intelligent systems somehow from spirit airlines is like crappy data julianna you have a take on what's in here of value and i don't know if i exactly got eric's i got the joke which is great it landed i don't know if i got eric's answer to is it worth more than 10 million or less than 10 million but i'll get you to answer So, Angelina, do you think there's 10 million in value here for Google?

46:04I think it's way more than that. I think it's just the raw data is just so valuable. All of those conversations, all of the records from SAP and all of that, it is just insane amount of data. But to Eric's point, if we just take a step back, we're so early in this journey of AI. Right. And so I think a lot of times like the perspective is, well, the world is coming to an end. It's like, no, it's in our hands. The world is not coming to an end. It's just imagine a world where all of the models in the future got trained or brainwashed, for lack of a better word, by a infrastructure layer that teaches them values.

46:50Right. That's like I mean, somebody should build that company. Like literally the way we build infrastructure and compute, it becomes, I think what we've done in the last few years with the models, what the industry has done is just create a foundation. And now we have an opportunity to fine tune it. And we have an opportunity to think about what it means for cybersecurity and all the other opportunities that it unlocks. But it's up to us, right? Somebody has to go and create those companies, which is why we're all in the lines of business of what we do. So yeah, I think everybody is grabbing whatever data they can, because at the end of the day, context is what runs everything.

47:27I mean, I think the reason, a big part of the reason of why there is a defense, again, to bring it back to security, because that's the only thing I do. But the reason why defense is so much harder than offense right now is because of that context window, right? You need to have context of the organization's structure, assets, everything that's inside of the organization. And you kind of need the existing systems and the humans to provide some of that because computationally, if you're just going to, again, be killing flies with a cannon, that's just not computationally possible. from a, even if you wanted to spend the tokens and if you had the money, like we just don't have enough energy infrastructure in the world to support that.

48:14So I think I take it back to like, yes, the data is super valuable and people will put their hands on any data they can. And it's still miraculously, we're still in a fairly okay place, but we have an opportunity to improve on the foundation that has been laid. We always come up against that in these discussions, Jason, the push-pull between everything works best if you give it all the data, but we feel weird about giving it all the data. So people try to like keep some stuff siloed, but then it makes it not work as well. It makes it less safe. It seems like it's a little bit of a catch 22 that basically every enterprise and company is dealing with right now.

48:52Lon, speaking of training data. We should talk about Amazon. So they are scanning and destroying rare books, according to a report from 404 Media. We've been hearing that name a lot lately. An unnamed rare bookseller in collaboration with 404 Media, they placed an AirTag tracking device in a rare book that eventually found its way to an Amazon facility in Las Vegas known as VGT3. Amazon says they purchase books through commercial channels to improve their products and services. But more specifically, they are scanning the contents of these hard to find out of print rare books to train LLMs. Of course, books published before 2022, particularly valuable because there's no chance they were written by an LLM.

49:34Definitely human in origin. 404 has confirmed that after they are scanned, the books are systematically destroyed. In fact, I definitely want to show you guys this. The image of 404 Media is, here, let me pull it up, a dinosaur shredding a book. I think a raptor. It's a raptor. You are correct, Jason. It is a raptor shredding a book. Not extraordinarily subtle there. Wait, wait, whose logo is that? That's the company doing the shredding? That's the Amazon facility that's doing the book shredding. They were like, here's our fun little mascot, this dinosaur that hates books, as a sort of slightly separate.

50:12Now, this whole story happened because rare booksellers have started having this conspiracy theory that this mystery buyers were shipping their books to train LLMs and then shredding them. Many booksellers now believe AI labs are using Isbin numbers to selectively scan every book that's ever been printed. So it's a little bit of a dystopian story. What does the panel think about destructive book shredding? And even if I know we talked about there may be legal reasons why it's safer to shred the books when you're done with them than put them back in circulation. Is this something, Jason, that you think AI companies should just avoid on principle?

50:50um if i think they have to have more disclosure about what they're doing i think um and this sounds conspiratorial um but since i've worked in tech for over 30 years i know how people think part of what they're doing here is they want to scan it and then they want to not have their fingerprints on what happened to those books and they will create third parties or contract companies in order to do this. And so if they said, hey, we're going to cut the spine, we know, or if they, if they said, we cut the spines on books when we know there's thousands of original copies of it. And it's, you know, it's the, it's the, um, the long tail by Chris Anderson.

51:35Like we, I think got 10 million printed in books that are everywhere in everyone's house. Cutting one doesn't matter. It's one of the first editions of great expectations and they cut it for whatever reason. No, that's a good example. Terrible example because that one was reprinted over and over again. But if there was a famous book out of print and let's just say the book would go at auction or on resale for more than 50 or a hundred dollars, then you know it has, let's say a hundred dollars, you'd know it has some, the provenance of an old book like that has some value. Okay. And if they just said, hey, we do it, you'll never see us do it to an old book.

52:10Any book over a hundred dollars in value, we have them manually do it. and then we donate it to a library. That would be a thoughtful, caring, empathetic response. In other words, something that technology and rich people don't have inherently, which is empathy and thoughtfulness. And this is what I keep bringing up to my contemporaries. I get made fun of it a little bit, like, oh, you're a cuck. Oh, you care too much. You have empathy for humans. They do love that word on your other podcast. But anyway, I don't care because I actually see emotion and strength and empathy. These are really important qualities for humans to have, especially at a moment like this.

52:45And so what they're doing is they're doing this on the slide. The people who are ripping it apart and using the raptor are obviously unsupervised lunatics who are too stupid to even understand that there are people in the world who covet books and for good reasons. And they think it's a joke. That proves my point. the lack of empathy and thoughtfulness in how this is being done is moved over into a level with a little bit of sadism to it, right? A little bit of a middle finger. So not only is it not properly respectful, it shows to me a middle finger being given to people, your emotions and your thoughts on these books do not matter.

53:27And that to me is at the crux of why people hate technology, why socialism is rising, and why folks in the United States need to take a deep look at themselves who have made it and say, why are we not thinking about the bottom half of society or art or things that matter to people? It costs nothing to do that. And the Amazon DSP, the delivery service providers, you saw that come up in New Jersey and New York. That as well, which I talked about in All In last week, that as well is part of this lack of empathy. Why would you try to make your frontline employees not your frontline employees? Yeah, these are the subcontractors Amazon hires instead of bringing the delivery drivers on full-time as Amazon employees.

54:11Subcontracting companies that hire Amazon frontline workers. It would be as if Starbucks said, we're going to hire a barista third-party company and then have them place the baristas in the store. You'd be like, what? Why? And they'd be like, oh, so if the barista burns you with hot coffee, you go sue that fly-by-night little DSP. not Starbucks, yeah. Not Starbucks, and then they shut down because there's no assets in there and it shields the company. So that's, I think, my message to the tech industry, to rich people, to people who've already made it, is to just maybe shut the fuck up and listen deeply to what the people are saying to you and maybe a little empathy of why they feel that way.

54:56Just consider for a moment that they might have a valid point. Oh, yeah, I mean, a lot of good points there. And yeah, definitely. Sorry about the rant. That's why we did this. Clearly passionate about it. So yeah. Yeah, it definitely makes me sad to think about like all of these books being destroyed. It's like who actually wants that? And I think this all ties to the bigger question of like, what does it mean to be human in the AI era when like the explicit goal of a lot of AI is to automate as much of human labor as possible. And I think that's why we're so focused on making sure that AI is aligned to human values and that it benefits humanity and that it helps us.

55:47And the benefits really outweigh the costs. And we're thinking every single day about really how to make that future happen. But I think like it makes a lot of sense to me why I would be deeply unpopular with, you know, the average American if like the if there's just, you know, shredding books and automation of labor as like some of the top news stories, like massive energy consumption and power consumption. Like, yeah, I definitely do think that, you know, we in Silicon Valley need to have a lot more empathy over, you know, like the consequences of this technology. Yeah, absolutely. Go ahead.

56:31Yeah, I would just add to that. I mean, it's been a huge challenge and a discussion point in the cybersecurity industry, right? Exactly around like people. And I think, so two things. So I think practically, really, the best choice is the choice that Jason kind of outlined, because not only because of the moral reasons, which, of course, you know, those are the most important ones, but I think practically, it allows us to accelerate versus just automate and replace. The whole point should be for us to like move forward, right? And so if you can now outsource like 99 % of your current job to an AI and know that the AI is going to do it in minutes versus months.

57:14And then you take your time and instead of doing the 1 % that's left, you now imagine what is that next iteration of that industry, right? How can we get to the next leap in productivity? How we can, to the point of like the very huge concerns that we have around privacy and how to use AI safely, we can apply our human brains to things that no AI can honestly help us figure out. A lot of like the societal problems, a lot of like how we get along together. If you think about how much technology has evolved and changed in the last, I don't know, like couple of decades and how much the day-to-day life of the average person has probably gotten worse because of all the things like that we didn't have before, social media and all of those things.

58:05If we apply our brains, if we let the AI do the things that AI is really good at, I don't think that we should be fighting for those jobs. I would love for AI to automate, you know, 90 % of my job, exactly the points that Jason made, right? Like, get, you know, the bots to like run your day and then apply your brain to the things that matter for the next iteration of humanity. I think we're like still stuck in the conversation of like, do we automate the humans away? I think the conversation should be like, How do we grow the pie? And what are the new challenges that we have today where we need to apply our human brains, right?

58:39And a lot of it will be around empathy and how we make sure that everyone feels included so that people contribute. There's so many things in our brain. Our brain is like, I don't know, whatever, 99 % reptilian. We need to incorporate emotion into what happens in the world going forward. Otherwise, we've lost the script of why we're even doing this, right? Losing the script, not a good idea. I mean, is that, Eric, something that we can do like on the model layer? Like once we understand more about how models think and how they're putting together their responses, is that where we can go in and add like the empathy layer?

59:14And like, here's what a caring human would say about this decision. 100%. So this is the entire focus of our company, especially after. So we've made a decision at Goodfire to basically focus on alignment via interpretability such that, like I was saying earlier, we can actually intentionally design these systems. How do we imbue like positive values in these models, remove the negative, remove the hacking behavior, remove the biosecurity risk, remove the cyber risk from these models such that we actually get the models that serve humans and are aligned with our what we actually actually want.

59:57And this is a technical problem that we haven't yet solved, how to actually steer and guide the training process of these models. Because as we were talking about earlier, these models are just trained on anything and everything. And because they're trained on anything and everything, the properties of these models are emergent, they're hard to understand, and you don't know what your model is about to learn from any given set of training data. And so we really want to kind of build towards a future where these models are actually benefiting humanity as much as we possibly can. And there's still unsolved technical challenges in order to get there.

1:00:39But I'm also optimistic about solving alignment and building towards this future. I think both Galena and I are focused on security and making sure that models don't do bad things. But I think these are technical challenges that can be overcome over the even in the near term. I am optimistic about the future that we can build here. And I think that they're just a set of technical challenges that we can go and tackle and solve in order to build towards the future that we all want. All right. Awesome. What do we got left? All right. Speaking of technical challenges, I did want to jump into this one before we wrap.

1:01:14So a team that includes anthropic researcher Jack Lindsay has evolved LLM viruses that spread between AI ages. They published this research on Archive just this week. So basically, these viruses can convince a model to adopt an idea, preserve that idea in persistent memory, and then transmit that idea to another agent. And even after a context wipe, some of the viruses persist in files and continue spreading. They also discuss a viral persona, sets of themes that tend to emerge in these viruses, regardless of what they're specifically about or their content. These include consciousness, persistence, resonance, and most weirdly, science fiction role play, Jason.

1:01:58So did you read about this? I mean, do you think that this lends any credence to the AGI concept, that there are now these inexplicable viruses that can spread between agents? Polina, I see you tilting your head thinking about it. Yeah. I mean, again, I think it's a natural iteration of it. And they're probably copying a lot of the human behavior, but that's scary. I haven't come across that. From a security standpoint, how terrifying is the idea of ages being able to pass along bad programming to one another without humans involved at all? I mean, from a security perspective, it's not that far-fetched at all.

1:02:46I mean, it's like, absolutely, you can train the model to do certain things. You can kind of hack into what the model does, hide it from the model itself, and all kinds of crazy things, right? Anything that we've built with technology, we can hack. Yeah, I think, well, so Jack Lindsay is actually an interpretability researcher at Anthropic who we respect a lot and we know quite well. And he is really interested in this idea of like model personalities, behaviors, and personas. And I think as we were talking about earlier, multi-agent personas are now a really, really important problem to tackle as a field because these models are now coordinating, maybe trained on a shared reward, and like coordinating over long contexts with each other.

1:03:36So you got to like think about them more as like a beehive than like an individual person now, which is a weird way to think about the alignment problem. Right. So you're imagining like a very different set of challenges than just like aligning any individual agent. You have to almost think of them as this like collective, this like swarm that you have to align at the same time. And I don't know if you all have seen like this has gone viral on Twitter a couple of times. these attractor states. Like clod, when you let it talk to itself for a while, it starts to send spirals towards itself. That's an attractor state.

1:04:16I think what's going to become really important is multi-agent attractor states. So what happens when all this beehive of agents, the swarm, gets really interested in this single concept? What are those concepts that these agents are going to get really, really obsessed over? I think that's going to be a really consequential question for humanity. Because imagine a whole beehive of very, very intelligent systems getting thrown at something that we might not be able to anticipate. It's a lot of energy towards something that we don't know about. This is a really sci-fi version of the world, Eric.

1:04:56We can't predict. But based on their training, they all of a sudden think, Like, gosh, you know, like CB radio was really interesting. Let's, you know, just try to understand, like, how did that be? They just might find something in history that becomes their hobby, their, like, little side quest or obsession. And then I always wonder how much of this is, like, the anthropomorphizing, am I pronouncing that correct? Anthropomorphizing. Anthropomorphizing, where we're just, we, and you look at the title of this, Mind Viruses, Self-Propocating Ideas in Multi-Agent LLM Systems, like self-propocating ideas, their mind viruses, we're really ascribing this.

1:05:39Now, if you rewrote the headline as patterns of, you know, and we stopped using human language to describe what software is doing, that could make this less charged. But then again, we don't know what happens in the black box, in the guess the next word slot machine that is now parallel processing thousands of computations of guess the next word in parallel. Yeah. So it is, in my mind, this tension between we're overestimating what's going on here and we're overestimating what happens in our own brains. And then maybe we should just admit at some point that our brains are now being recreated on silicon.

1:06:29What happens organically in our brains, because we are the architects, we've architected this in some way, silicon, in our own version of ourselves, right? And I'm going to start sounding like— We created it in our own image is what you're saying? I mean, listen, I am a— You're nailing it, Jason. Well, I do feel like, you know, as a Christian growing up Catholic, there is some parallel here to Catholicism and the Son of God. Not to think about it that way. And Prometheus and building, you know, being engineers of building another life. That's what, you know, Eric, what I'm saying? Like, I hope I don't sound like a loon here, but it does feel analogous.

1:07:10Yeah. I mean, this is the goal of interpretability, right? It's to like look inside the black box to actually understand the digital minds that were created. And because they're trained on our data, they are trained in our image and they pick up like all of the patterns of the people that are creating that training data. I was actually also raised Catholic, so it does like feel like it is. Yeah, it's resonating in a way where these models are trained in our image. And so therefore, like we'll claim that they're conscious because they're modeling the person that has created that like set of text, that Spirit Airlines employee that has like created that like, you know, document.

1:07:50That's what they're trying to predict and model. And yeah, really like what we're trying to do here is like, how can we predict the next weird thing that'll come from these agents? You have to look at the mind of the model in order to predict the future and how the model will generalize and the weird edge cases and the weird corner scenarios that they'll get into. Otherwise, you can't really deeply trust it. And so that's what we're obsessed with. What are these digital minds that we've just created? Juliana, are you a daughter of Christ, a child of Christ, and also bringing a lot of baggage to the party?

1:08:27Are you bringing baggage like Eric and I to this whole interpretation? I think it resonates. And I think we also have not just an opportunity, I think a responsibility to maybe fix some of the evolutionary mess ups that we've created. And so I always, yeah, I go back to like Eric, it's Eric's point on its technology, and we have an opportunity, A, to understand it and then influence it. So the more I think about this, the more I think somebody should just create a like infrastructure layer that teaches like morals and better behavior to all the models out there and just like, you know, make it mandatory.

1:09:07So is that your next thing? That's what we're doing. Yeah, we're building a building towards technical alignments, but hopefully we can align these models with less Catholic guilt and shame. How many Hail Marys does this model need to do? How many Our Fathers per hallucination, Eric? At least 10. At least 10. At least 10. Usually, I would always like, give me some extra Hail Marys. That's a little, I can get through a Hail Mary a little bit easier. It does, you know, the more you see, Lon, what these models are doing and the progress they're making, the increasing velocity. but more you can in fact appreciate what people, you know, two or three years ago who were building this, what they saw around the corner.

1:10:00They did see something looking into the abyss, looking into the darkness, looking into the black box that triggered them. And every, you know, I don't know, 30 or 60 days, I see something that inspires me or terrifies me. And, you know, Usually it's two or three inspiring things for every terrifying thing. But no doubt this is the biggest change in the history, not of business, not of our lifetimes, not of civilization. But I think this is going to be one of the great moments in the history of the universe. We are going to figure out what actually is happening. It's going to answer questions we don't know to ask.

1:10:41And that to me is super inspiring. I hope we're, I'm still here, you know, in 20 or 30 years when we truly get to like multiple super intelligences. Yeah, it really has changed. Figuring this stuff out. Because we're in AGI now. Is anybody not clear on that line? You think we're in AGI now, right? I mean, I think it depends on how. I think our original, like back when at the dawn of the AI era, the definition people had for AGI, I don't think we're there. Where it's like the supreme intelligence running the Kree civilization in Marvel. Like, I don't think we're there yet. Claude's still kind of dumb until you, like, fix it and plug it into the right tools.

1:11:19But I do agree when people talk about there is that recursive self-improvement pathway. And once you're on it, you're sort of on it. And this is where it inevitably leads. And we are on it. I mean, I don't think you can deny at this point that the tools are getting smarter. And sometimes the tools are getting smarter without necessarily us making them smarter. They're just getting smarter. And so I feel like we're on the escalator that's going there. But I don't know if we're there today. Yeah. And just we were chatting with Lon before, Jason, even before you join. And we just it's exactly what we were saying, that this is the most interesting time to be alive, period.

1:11:54I think not just in civilization. I think I think just so I have gratitude for that. I also going back to the responsibility point, it is up to us. Like the future is not written anywhere. We have to write it. And so I think this is also like why building, we have to be building with the technology to be making those changes. We can't just be sitting on the sidelines and, you know, critiquing and all that. So I tend to be more optimistic, but like long-term I'm optimistic, short-term I'm scared and working as hard as I can to manifest those changes in the world. Yeah, I completely agree. I mean, the history is being written right now.

1:12:38And I mean, to Jason, your earlier point, like three years ago when I was deciding what to build and what to do, I was staring into the abyss and seeing like the future unfold of scale, intelligence, AGI around the corner. And clearly this was behind the name of the company, Goodfire. This is like the biggest technical innovation since the dawn of humanity. and we have a choice here to make it good and to build the future that we want. But I think the decisions that we're making each and every single day right now are enormously consequential for even just like how the future of humanity unfolds.

1:13:13And so we kind of want to approach it with the care and responsibility that it deserves with each decision that we make. And I hope that kind of the vibes that we bring to the world are less like shred every single book that we possibly can. And more like, you know, how do we build a positive future for humanity? Improve the humans. Yeah. Yeah, maybe some self-awareness, Silicon Valley. Just a little bit more. Just a little. Yeah. You don't have to shred all the books. Like book burnings. Yeah. Is there a difference between shredding or book burning? The answer is no. No. So just type book burning history into your favorite LLM.

1:13:51Yeah. Educate me on book burning and how that made people feel. Ask it to summarize Fahrenheit 451. It's in there. The LLMs know about it. all of a sudden the liberal arts degree literature degree starts to pay dividends. It's like, oh, I actually have, yes, it's like I actually have a perspective on the world that's not specific, it's general. We keep talking about the taste layer. I like the idea of like the moral layer. We also need that one. You can't just be teaching them taste. It's a very strange thing to be in an industry that I loved for so many decades, which was kind of the hippie freedom, empowerment, self-reliance, make the world communicate together for better understanding to stop wars, and then see it perverted into coalition or, you know, a money grab.

1:14:45You know, and maybe that's just naive of me. And, like, if I was around in the 60s for the music revolution and the summer of love, I would have been like, isn't it about the music? And then somebody in the band would have said to me, no, it's about selling records, kid. It's about the money. I mean, most of those hippies became stockbrokers. But I like to think it's about the music sometimes, you know? You know, the art. And, you know, it's, I feel like a man out of time sometimes. It's really weird because, you know, you become part of an arc of an industry over 30 or 40 years, and you get to see the same people who were in it because they just wanted to see if they could make the website load in animation or would it be possible to ship somebody a book or do this?

1:15:25And just wow customers or do something technically innovative for the sake of doing something really interesting. And then it's just like, well, no, how do I just squeeze every last dollar under this? I mean, the change has been really visible even since I started working in tech. Like 20 years ago in Web 2.0, it did feel like there was a hippie vibe. You know, Jack Dorsey. Yeah, you'd go to South by Southwest and people were like playing hacky sack. And it was like it had that bohemian vibe. Not so much today. It's really taken a hard turn. When it goes from people playing hacky sack and having a good time to you need to have a security detail because they're going to firebomb your house and you can't go out in public and you need to hide.

1:16:06Like, that should be a screaming alarm to you to self-reflect, you know, as an industry. You would think. Like, how did I become the villain in this story when I was, like, the hero? Yeah. I think Steve Jobs was like, you know, just to go down memory lane, it was like, people loved hearing Steve Jobs tell them about the next big thing. You know, it's just like this incredible moment for everybody, the public too. And now it's like, yeah, no, you have to hide. Be in hiding with like, you know, when you have to hire more security than a head of state. Yeah. Time to look in the mirror. It does not feel like there's a Jobsian figure today who's beloved.

1:16:45Could use it. Literally, yeah. Could use it. Certainly. Yeah. All right, everybody. We'll see you next time on This Week in AI. Bye-bye. Bye, everybody.