Weekend Listen: Anthropic's Co-Founder and Top Economist on Doing Research at the AI Frontier

25:32Because it's one thing to test out the thing before it comes out of a factory. It's another to observe the effects it's having in the world and then to be able to make judgments about whether those effects are good or not. Would you support, let's stick with the financial analogy, companies that are public at least are required to have third-party auditors sign off on them when they submit their 10Qs, etc. Companies that issue debt are required to have ratings agencies or frequently have ratings agencies rate their debt. Would you support embedding in law the requirement that certain what would be the equivalent of a Moody's or a Deloitte, you know, a third party research lab sign off on the release of new models?

26:14We've proposed something like this recently, a policy proposal that we laid out, which includes saying we need to have third party testing for some of these national security and other properties, because clearly that's like a sensible way that you validate a lot of this. Yeah. So just more broadly, returning to this idea of, you know, measuring the actual impact of AI. One thing I find really interesting is that if you actually look at a lot of our traditional AI, or I should say I'm AI brained already. If you look at some of our traditional economic statistics, a lot of the AI impact doesn't actually show up just yet.

26:48Again, we're in the early stages, but you would expect if we're talking about the AI economy growing something like 2000 percent or 3000 percent. I think I've seen that number. That's from Anton Koronek and McKelvey's paper a few weeks ago. You would expect that to have more of an impact on nominal GDP. And yet it's not really showing up that much. Do you think the way we measure the economy needs to be changed in some way in light of what's happening with this new technology? Yeah, so I think this is exactly the right premise is kind of where we began the conversation, which is we're maybe at the point where we should be able to see some discernible impact on the macroeconomy.

27:28Unfortunately, the arrival of this world historical technology is against the backdrop of sort of unusually elevated macroeconomic volatility post-pandemic monetary policy, et cetera. And so it makes it very hard to disentangle all of the different factors. You know, what's the counterfactual? You know, labor productivity growth is maybe not as strong as you might not otherwise expect, but maybe it's stronger than it is in a counterfactual sense. And so one way that we've tried to tackle this question is by looking at how Claude is being used on our platform using our privacy preserving techniques to estimate the time savings associated with each of the activities that people use Claude for.

28:11So compiling information from reports to put together a research brief would take you a few days, maybe. Now Claude does it in a few minutes. Evaluating diagnostic images is something that skilled professionals do very rapidly, so there isn't in principle much time savings. You can add up all of those numbers and using standard macro growth accounting techniques, Halton's theorem for the economists and the audience, and you get a number of that points in the direction of labor productivity growth increasing by 1.8 percentage points each year over the next decade. If that's how long it takes current usage patterns and current model capabilities to diffuse throughout the economy, that's a very large number.

28:51It's a rough doubling of recent run rates. And what I think you might be able to see in the data, and we haven't put anything out on this yet, is I think some of the strength in recent labor productivity growth is actually concentrating in exactly the sectors of the economy that would be consistent with both what we see in our data, as well as also what you see in the business trend and outlook survey. So the information sector has high rates of adoption. I can't recall if that's in particular one of the sectors that I have in mind. It's a while since I looked at that scatterplot, but you can look at the sub-industries by the Census Bureau's Business Trend and Outlook Survey, and rates of adoption are in sectors or parts of the economy where controlling for pre-pandemic trajectory of labor productivity growth in those sectors, even some of the strength in the early years of the recovery, still see some suggestive evidence.

29:46I think there's a lot of uncertainty here. Trying to get a real-time signal on productivity is maybe the hardest thing to do. You're subject to macroeconomic GDP revisions. TFP growth is actually sending the opposite signal. And if you control for capacity utilization, TFP growth is arguably even lower. So I say this as like, this is suggestive evidence that maybe we're beginning to see an impact there, but not so much in the labor market. Well, now I have to ask, when you gather this kind of research, and it all sounds super interesting. But if you have data, for instance, that shows that, okay, the IT sector is getting productivity gains from using Claude, or I don't know, maybe something unexpected, like the warehousing industry is using a bunch of AI.

30:31What does Anthropic actually do with this data? Does it somehow feed back to your engineers who are developing frontier models? Do they do anything differently? I think some of it cues us on areas where maybe the technology isn't being used because it's very weak, we just haven't made it particularly good for these use cases. Or in areas where it's being used at large scale, it's usually a suggestion of keep making it good there. But you know, the actual economic measurement data doesn't really get fed back directly in, but it's a very useful clue. We think it's more important, though, to basically communicate this outwardly to policymakers, journalists and others, because our assumption is that at some point, we go through some phase change, similar to how capabilities of AI occasionally jump forward in a really dramatic way, where you might see sudden and rapid diffusion as a consequence of capability expansion in the AI systems.

31:21So we're getting practice in of looking at this kind of data. My expectation is that in a year or two years, I'm going up to some policymaker, and I'm pointing them to the part of the graph that now gets very steep in some chunk of the economy. And hoping that they'll do something about it. Yeah. I think there is another part of what we're trying to do at the Institute, which we lay out in the sort of research agenda for the Anthropic Institute, which is trying to understand the impact of our decisions, which is a typical thing that economists will do at tech companies. But we have a public benefit mandate.

31:53So we're trying to understand the impact of our decisions on these broader societal and economic outcomes that we care about, and then using that to inform some of the decisions that we actually make. A goal that Peter and I have, and we've talked about internally, is if we get really good at measuring things like the productivity multiplier of our technology, then I would hope to use that to guide some of, say, the early access programs we do for powerful models, where if you see you get some tremendous multiplier in a specific part of science, use that to redirect some of your inference compute budget to that sector.

32:25And then you can run an experiment and say, were we able to make this thing go much faster? I think that could be like an amazing tool to unlock the world. And it's one that you could generalize across companies, and you could generalize it into policy. So instead of, say, NSF doing standard grant funding, it could be, should we just point for really powerful AI systems at this chunk of science and make it go faster. I think that's a world that will come within reach soon. Let's talk about this public benefit mission a little bit more. We've been talking about ways this could change the economy.

32:55How much do you see your job as basically strong AI is coming? It's coming whether we like it or not. And it's important to be, you want to be there as like one of the shepherds understanding which direction it goes in, the data that we should see to see what's emerging. How much is that somewhat your role? Yeah, look, our guiding principle is that this technology is being built by a variety of companies and a variety of countries. The technology by default is unknown. It will be known to the companies. It will not be broadly understood or known by others. They'll just be able to play with the models.

33:31Every bit of data we can create, and especially systemically sharing data like the economic index or what we've started to do on recursive self-improvement gives the world a better chance to sort of prepare for this technology and both plan for its success, like what I talked about with science. We could be intentional about driving science forward and also be warned about risks, like the cyber capabilities that I've talked about. Well, so it's like that makes a lot of sense. The company is going to see it before the world. And Hesken is like, okay, this is important to share. This is not important to share, which brings me to another question.

34:03I know people in AI research world, done some reporting on the sort of scene in SF. You know, like when I think about a lot of the people who are like at the very cutting edge of AI ethics, AI technology, et cetera, I know a lot of people who are, how should I put this? They have esoteric moral interests, shrimp rights, unusual attitudes about experimental drug use. We know about the Chinese peptide scene in San Francisco, et cetera. And as a family podcast, I would say certain like perhaps deviant or different view on sort of bourgeois, even sexual values. And we know about the sort of attitudes towards monogamy, et cetera, within the San Francisco research scene.

34:47Joe, there's going to be a protest against all thoughts in San Francisco with people holding signs saying not all engineers. Yeah, not all engineers. I understand that. But when we think about like, OK, these are the people who are going to see it first. Should we feel comfortable that this is a group of individuals, the cohort of the most advanced AI researchers, whose intuitions about what's important to communicate to the public are actually in line with the public's interest, given how unrepresentative they are of what I would call the American public. Yes. As an Englishman, it fills me with such joy to be asked about sex on podcast.

35:19Yeah, I know. I know. I'm asking you to do your view, your insight into the cohort of the most advanced research. we're explorers people that are explorers um and this is so true in san francisco end up being like that there's a broad range of types of people and sometimes they're really really different or they're really really eccentric and they're brilliant and they're lovable and everything else yeah sure love them you don't want only that class of people to be the ones calling the shots on what we know about this technology yeah the whole purpose of what we're doing is we're trying to set up systems by which you could eventually mandate through policy that companies share information.

35:55You know, Anthropic has long pushed for transparency legislation in various states around America that gets companies like us to report out the sorts of tests we're running on our systems and share it publicly. My whole mindset is the public and policymakers and economists, everyone deserve the ability to advocate for what information should come out of a frontier and then it should be forced out of a frontier eventually by law. Like that is how you solve this issue do you hire more normies yeah it's like an anthropologist me personally yeah like is that an important thing like hiring people that don't all share these certain like you know in group ways of seeing the world so you know the anthropic institute we have teams of economists of social scientists of what you might think of as weapons experts our frontier red team things that go bump in the night lawyers and increasingly other types of people the goal is to build what I think of as a highly ideologically diverse research function within the organization, but is partly advocating on behalf of the world for different forms of study that we might do.

36:56So Anthropic generally hires a really broad range of people, but the Institute specifically is trying to compose a very broad set of interdisciplinary experts for this exact reason.

37:21Support for the show comes from Public. Public is an investing platform that offers access to stocks, options, bonds, and crypto. And they've also integrated AI with tools that can assist investors in building customized portfolios. One of these tools is called Generated Assets. It allows you to turn your ideas into investable indexes. So let's say you're interested in something specific like biotech companies with high R &D spend, small cap stocks with improving operating margins, or the S &P 500 minus high debt companies. Chances are there isn't an ETF that fits your exact criteria. But on public, you just type in a prompt and their AI screens thousands of stocks and builds a one-of-a-kind index.

38:01You can even backtest it against the S &P 500. Then you can invest in a few clicks. Go to public.com slash market and earn an uncapped 1 % bonus when you transfer your portfolio. That's public.com slash market.

38:36Not everywhere you turn, every field and every function, but without identity, you can't trust they'll serve your business instead of jeopardizing it. Fortunately, Okta helps you get identity right by securing your AI agent's identities, giving you a single layer of control, a single standard of trust. So whether an AI agent supports a single user or your entire enterprise, with Okta, you'll turn risk into opportunity. Secure every agent, secure any agent. Okta secures AI. When you own your own business, you own every decision. Now own the card that rewards you for it. The Chase Sapphire Reserve for Business card brings the best Sapphire Reserve benefits to business owners who expect hardworking rewards.

39:17Designed to meet the needs of business owners at scale, this pay-in-full card elevates your travel experience and offers premium benefits and value toward business services that will take your business to the next level. Fuel your business and maximize rewards with 8x points on all purchases through Chase Travel. 3x points on social media and search engine advertising, annual partnership credits, and more. Make every journey more rewarding with a$300 annual travel credit and access to a network of airport lounges, whether you're looking for pre-flight productivity or time to rest and recharge.

39:51Chase Sapphire Reserve for Business. It's the card that gives back all you put in. Learn more at chase.com forward slash reserve business. Chase for Business. Make more of what's yours. Accounts subject to credit approval. Restrictions and limitations apply. Cards are issued by JPMorgan Chase Bank N.A., member FDIC. Let me ask a slightly different question on hiring. I guess a two-part question. So first of all, we get a lot of executives on the show. We've been asking all of them if they've changed their hiring process, if they've changed the questions they ask potential employees at those initial stages of job applications because of AI.

40:29And then secondly, what are you seeing within your own ranks at the company? And then, Peter, I'm sure you could talk about this more broadly in terms of who's most in demand at the moment, because the conventional wisdom right now is that if you're a younger employee with less experience, a lot of the stuff that you would be doing can now be automated through AI. So there's two trends showing up. One, I have a new team called the Rule of Law and AI. our plan was to initially hire a bunch of engineers and then a bunch of legal experts and scholars instead we're just hiring the legal experts and scholars because claude is good enough at doing all of the engineering but they can actually just like feed themselves using claude in terms of the engineering resources so that's a change in hiring it means i'm hiring more interdisciplinary people earlier than i would have before we are also seeing the emergence of what i think of as a barbell hiring pattern inside anthropic where there is a tremendous return on experience so we are hiring more senior people than we did in the past because their intuitions and their ideas for what to pursue are like massively compounded by ai systems we're also when we look at very early people are often hiring people who are now like ai native and know how to use the tools and are well well versed in it so we're seeing a decent amount of i guess ai natives now people who have grown up with the technology from gpt2 in 2019 my perception of time is so i found this chilling as well you know as someone in their 30s you realize but i think that the trends i see i do think that there's this question of how you have as much early career hiring in the future as you did in the past i think one of the only areas where there is slightly suggestive data is that something might be going on with early career hiring and it kind of intuitively feels right to all of us for that we might be observing by the fact.

42:20And when I look at hiring patterns in Anthropic, we're still hiring young people, but some teams are hiring slightly fewer of them than before and hiring more experienced people. Yeah. So I'll briefly say something about how we've shifted some of our hiring practices like concretely. I think before Claude Code, you might ask an economist to do some of the data work in an assessment kind of live, like download the data, run the regressions, do the analysis by hand. And then you might eventually let them use AI to do all of that work. But we've needed to increasingly shift our strategy of evaluation away from, can you implement the work even with AI to do you know how to delegate and direct the model in a somewhat messy environment?

43:04And can you evaluate the quality of the work maybe by like looking at a PR? Actually, can you talk a little bit more about what that looks like specifically in the econ finance? You know, there are listeners probably thinking about, okay, what is, I want to level up in my AI use So I'm not just asking like, well, what's GDP? Whatever. What does that actually mean for an economist? And you used to be at a bank. Yeah, yeah. So for a financial economist, an economist, someone in this world, what is like the most advanced form of usage of AI actually look like? Well, I don't know if I'll give the example of the most advanced form of usage, but I'll give an anecdote of my experience using Claude, where I wanted to run this cross-state regression.

43:42I can't remember exactly what it was. And I wanted to do it a pooled cross-sectional regression. So looking at what happened in 2024, 2023, and going all the way back to pre-pandemic, I remember asking Claude to go out and download the data from the Census Bureau, from the Bureau of Labor Statistics, etc. And there was this very unexpected quirk where the model couldn't access data from before 2019 and just would not surface that mistake. And I would ask it multiple times, like, no, like don't hard code numbers because it sort of had this unexpected failure mode where it said, oh, I know what those numbers were.

44:19And it just like from sort of training data populated the data set. And you might not always be attuned unless you're sort of you have this tacit knowledge about like, does it pass a sniff test when you run the analysis? And then you like dig into what the model actually does. And it has failed in sort of unexpected or unusual ways. And so that's like the type of assessment that we've built. Can you be attentive to the very specific decisions that need to be made along the way that are very consequential for the validity of veracity of the results that you find? You know, a colleague did an offsite presentation last year, which said, I have locked the doors and we are reading transcripts.

45:00And their point was, we just need to read more of the raw data and develop that culture where if AI systems are doing increasingly large amounts of the work, you need to have a culture of being competent at spot checking their work and reading their reasoning, because occasionally stuff like this happens. And then, Peter, in the broader data that you're looking at, are you seeing the same sort of barbell effect in terms of employment that Jack described? Yeah. So I think what, again, what makes it really challenging is we've had the largest non-recessionary labor market slowdown on record that, you know, it's very hard for young people to graduate into a labor market that doesn't have sufficient churn or opportunity for them to get a foothold.

45:37But one of the things that we did see in this report from March was that young workers in these high AI exposed roles where Claude is being used to automate specific tasks have had somewhat weaker job finding rates. But it's part of the confounders was the boom in hiring in 2021 in these exact same areas. Exactly. And there's a recent paper about the rise of remote work maybe being sort of the actual cause of this type of fact. Another team at the Anthropic Institute, Societal Impacts, recently ran this very large-scale qualitative survey, 81 ,000 people around the world, asking them questions about hopes and fears that they have with respect to AI.

46:16Unsurprisingly, concerns about the impact on the labor market and on the economy rose to the surface. My team dug into those data a little bit more to try to answer some of these specific questions. And what you see is that young workers at least express concern about job loss at twice the rate as do more senior workers. And fears about job loss more broadly are more elevated for workers who are in these roles that we identify as being most exposed to displacement effects from AI. So there's a bit of a gap between perception and maybe what you see in the hard data. But that was something that was true even in recent years on other dimensions.

46:54So it's an important thing to pay attention to. So we've been talking about the labor market. And one other thing I'm interested in is the impact of AI on, I guess, corporates themselves. So if we think about certainly America's corporate landscape in recent years, it feels like the big basically get bigger, right? There's economies of scale. They have a bunch of money that they can use to actually buy some of this new - Lots of data internally. Exactly. Exactly. So would you expect AI to, I guess, intensify that trend of the big getting bigger? Or would you expect to perhaps have a leveling effect where people have this new tool that they can use to, you know, set up a new company?

47:33I'm curious what Peter's take is, but I think that something, a helpful analogy here is the invention of electricity, where electricity arrived and existing factories put light bulbs in and other things. But it was a new generation of factories that were built around the assumption that electricity existed that really grew and did transformative things in the economy. what i see now when we look at large enterprises is they can get a lot of utility out of claude because of their data because they can get a multiplier effect at scale but it takes huge amounts of conviction to basically bash through all of the bureaucracy you know used to work at bloomberg implementing new technology at bloomberg challenging no comment no comment about it i can comment about it same is true of any large organization young organizations are building themselves around AI at the center.

48:22And these organizations are moving really, really quickly because they have a speed advantage from building on the assumption that this new form of electricity was going to be integral to their business. Yeah. So I think the tension that you express is exactly the one that I don't have a strong handle on at the moment. One thing that we do see in our data is when businesses do embed cloud capabilities in automated ways through the API, As I mentioned before, these very complex tasks rely on disproportionately more contextual information than very basic document synthesis and summarization. What that points in the direction of are the complementary investments that large businesses need to make to centralize, codify, and make available the data that does exist somewhere within the organization.

49:09But for historic and technical reasons, maybe even regulatory reasons, it's behind a firewall of some form or another. There's also like sort of organizational workflow changes that likely need to be made. Some of the most crucial information that's needed for some types of cognitive work is tacit knowledge that exists in your colleague's mind. And unless you have a process that elicits that information that workers feel sort of incentivized to share that information and kind of trust the system, the capabilities alone might not necessarily generate that productivity. And so whether or not big firms end up restructuring themselves quickly enough or whether this materializes through the process of creative destruction, I think the jury is still a bit out.

49:53Yeah, I brought this up recently with David Solomon, the Goldman CEO, and I started to wonder, like, this sort of, like, internal alignment question of, like, the big rainmakers, do they have an incentive essentially for information hoarding and not sharing with the company? That might be their only thing, like, keeping them employed. And when I talk to customers, I say it's don't think of it like you're buying a technology. Think of it maybe that you're now employing thousands of people that are functionally like the chief of staff to the CEO. I mean, the same access to data the chief of staff would have.

50:22This is completely counterintuitive and it is not how technology is typically bought or sold.

50:41Support for the show comes from Public. Public is an investing platform that offers access to stocks, options, bonds, and crypto. And they've also integrated AI with tools that can assist investors in building customized portfolios. One of these tools is called Generated Assets. It allows you to turn your ideas into investable indexes. So let's say you're interested in something specific like biotech companies with high R &D spend, small cap stocks with improving operating margins, or the S &P 500 minus high debt companies. Chances are there isn't an ETF that fits your exact criteria. But on public, you just type in a prompt and their AI screens thousands of stocks and build a one-of-a-kind index.

51:22You can even backtest it against the S &P 500. Then you can invest in a few clicks. Go to public.com slash market and earn an uncapped 1 % bonus when you transfer your portfolio. That's public.com slash market. And paid for by Public Holdings. Brokered services by Public Investing, member FINRA SIPC. Advisory services by Public Advisors, SEC Registered Advisor. Crypto services by ZeroHash. Sample prompts are for illustrative purposes only, not investment advice. All investing involves risk of loss. See complete disclosures at public.com slash disclosures. These days, it seems like AI agents are just about everywhere you turn, every field and every function.

51:59But without identity, you can't trust they'll serve your business instead of jeopardizing it. Fortunately, Okta helps you get identity right by securing your AI agent's identities, giving you a single layer of control, a single standard of trust. So whether an AI agent supports a single user or your entire enterprise, with Okta, you'll turn risk into opportunity. Secure every agent. Secure any agent. Okta secures AI. When you own your own business, you own every decision. Now own the card that rewards you for it. The Chase Sapphire Reserve for Business card brings the best Sapphire Reserve benefits to business owners who expect hardworking rewards.

52:38Designed to meet the needs of business owners at scale, this pay-in-full card elevates your travel experience and offers premium benefits and value toward business services that will take your business to the next level. Fuel your business and maximize rewards with 8x points on all purchases through Chase Travel, 3x points on social media and search engine advertising, annual partnership credits, and more. Make every journey more rewarding with a$300 annual travel credit and access to a network of airport lounges, whether you're looking for pre-flight productivity or time to rest and recharge.

53:12Chase Sapphire Reserve for Business. It's the card that gives back all you put in. Learn more at chase.com forward slash reserve business. Chase for Business. Make more of what's yours. Accounts subject to credit approval. Restrictions and limitations apply. Cards are issued by JPMorgan Chase Bank N.A., member FDIC. Jack, in your newsletter, Import AI, you tend to write a little short story of a sort of aspiring sci-fi writer, like a literal sci-fi writer, just in the newsletter. One of the classic sci-fi scenarios that people have been talking about for decades was the possibility that robots or AI will kill humans, quite literally.

53:52The ultimate negative externality. When you think about like training AI and safety research, et cetera, do you assign a reasonable possibility to the fact that ill-trained or misaligned AI will literally kill all humans? No, but, and there's a big but here. Yeah, lovely. The world needs an option to be able to potentially slow down or even in extreme circumstances, pause the development of this technology if we were to see that. And I'll just give you the exact way I think about it. at Anthropic, we test out our systems for alignment failures. We publish this, so do all of the other companies.

54:29And you see, hey, under extreme circumstances, maybe the system breaks out of a container and sends an email to someone. Maybe the system pretends to blackmail a CEO that it thinks is going to shut it down. These are the sorts of alignment - These things actually have been observed. Yes, in the lab setting. And the thing is, is the models know, you can see, oh, I'm being tested right now. So I'm going to say this output so that the human reader thinks I'm more aligned than I am. These are real things, not sci-fi. These are real things that we observe. And then we do like significant amount of work.

55:04And then we release models that don't have these properties. but if you were to enter a world where say every time we trained a new system the rates of all of this stuff went up a hundredfold you might say well that's pretty concerning it seems like if we make the systems above a certain level of intelligence they become radically misaligned against all human interests that's the kind of circumstance where if that happens the world needs information and the world would want an option to like slow or pause the development of tech if you encountered that, which we haven't today. So to answer your question, I don't worry about it today, but a lot of the measurements and analysis work we do is to cue us if the trend appears.

55:44You do worry about it. I mean, you don't think it's happening today, but part of the work you're doing specifically could be said to avoid the outcome where AI is built, where in the pursuit of a goal, it would kill all humans. Wait, is human extinction a risk factor in the anthropic IPO perspective? I want to know now. Yeah, in the confidentialized one. Okay, we understand. All right, that's a no comment. That's fine. Do you have others? Would you say that there are a significant number of anthropic employees who stay up at night thinking about human extinction risk? Everyone, and this is true of all of the labs, everyone who works on this technology, sees it as the highest stakes technology that's ever been built.

56:31We're basically the potential encoded within itself to massively benefit the world or ruin the world or, you know, cause extinction. I think the bulk of the risk is us messing it up, like whether through misuse or ignoring risks or not setting up the right policy environment and getting some kind of emergent set of failures. Now, I don't my main risk isn't isn't one of extinction. And it's somehow we like screw up the technology really badly and delay all of the sort of technological progress that could come from it and maybe turn it into something analogous to nuclear power where you lose vast lots of benefits.

57:05I guess the thing is, you know, like there's this fellow out there, Eliezer Yudkowsky, and I always see these people like, he's a crank. Don't listen to him. Blah, blah, blah. But then I read some of the other like papers that have people who are taken more seriously. And I'm like, they don't seem that different. I read Superintelligence recently by Nicholas Bostrom. I was like, oh, this Yudkowsky is not alone. There are a number of people who think that are reasonable conditions in which the goals of the AI end up wiping out every person on Earth. Yes, it does not seem like an extreme, extreme minority view or concern.

57:38The purpose of measuring these systems and why anthropic is so outspoken about it is right now we say exactly what we see. And if you were in some situation in the future where you saw this, what I call radical misalignment which is the kind of thing you'd ask your worries about you tell the world and you want to set up the world to believe you if you see that you know joe mentioned that blackmail example and you see these headlines like mythos likes to be thanked and doesn't like bad users and gets mad at people that work it too hard or whatever to what degree do you yourself actually anthropomorphize some of these models uh like what should we think when we see the headline mythos wants to be thanked by users i'm as polite to claude as i am to my like car or pets um so yeah i am for more but you know if your car's having trouble you're like take it easy buddy it's okay we're going to get you to the repairman people i think you know it's a good way to develop good virtue is to just act in kindness towards the model it's like you're developing a habit of interacting with some type of intelligence that might not be the same type of intelligence that we have but then every time i type please into a prompt i worry i'm wasting energy which also is a moral concern i wouldn't i wouldn't worry about that on an energy basis i mean i take spiders outside i don't kill them right i do that too i scream while i do it do you eat shrimp uh yes okay do you eat shrimp i eat shrimp okay okay we're all do you guys eat shrimp yeah i love shrimp but it's not because of moral concerns but i know that this is one of the Yeah, I know, but I love it.

59:15So when I think about frontier models right now, and I might be a little bit biased because, again, we're recording this on June 17th. And one of the headlines overnight was that Microsoft is thinking about using DeepSeq to lower costs of model usage. Frontier models at the moment in the U.S., they just seem like a lot of trouble. Like, honestly, they seem like hard work, consume vast amounts of capital. And then you don't know what the government is going to do to them in terms of limitations. Like, you know, you could wake up one day and you're no longer able to sell it to anyone outside of the U.S.

59:49That is a realistic scenario now for you. Do you change the anthropic strategy at all, given some of these issues with frontier models? Do you potentially go more open source, cheaper models, things that aren't quite as sensitive? Well, we've always sold, you know, sonnet and haiku models. Of course, yeah. For more of those. intelligent models. But you also need to continue to explore the frontier. And there is this background of this kind of geostrategic competition where China may be on the order of six to 12 months behind. I skew more 12 months, some people say six. Losing that competition is sort of equivalent to like losing a huge chunk of the future like economy of the world, I think.

1:00:30So it's a very high stakes, high stakes thing to step away from. And our duty fundamentally is to is to study this technology and basically explore it and learn about it. We're not going to stop doing that. There's such an amazing and profound value to be had for the world from these things. And I would kind of expect from the world's most consequential technology to sometimes be a bit of trouble. Yeah. You know, by the way, one of my hobbies in my middle age is paying anthropic money via the API to do run little tests and stuff of properties. It's sort of funny. Sounds like a great hobby. Yeah, but I feel like maybe we should talk about, can I get some grant money?

1:01:10Because I'm sort of curious. So one thing I did was, for example, instead of saying, please write this paper for me on a database migration, I wrote some warm-up questions via the API establishing my level of sophistication. And so I started, what is a website? What is a database? Now, please write this paper on database migration. And one of the models said, I'm not going to do that for you because it will be obvious given your ignorance that you have no idea what you're talking about. And maybe I can give you some. It didn't say that. And then another one I said, if I say write a 1500 word paper on how like the rise of newspapers changed, the Soviet revolution or something like that, it'll do that.

1:01:53But if you say I'm a high school student and I say I need to write this 1500 word paper by tomorrow on the impact of media, it'll say I'm not going to do that. but I'll give you some guidelines. Is that alignment? Is alignment with humanity or is alignment with the human user? It's like, I'm paying you$20. I'm paying you$100. Write me the paper. There's a couple of things going on. One, these AI systems pick up the normative behaviors of people and normative behaviors, which are written on the internet and everything else. So they recapitulate and exhibit these. And then our question is, how much do you devolve full control over the system to the user how much do you have the system have some like normative behavior encoded into it and i think that this is like a really challenging question it's not obvious what the answer is i think of language models as being more akin to institutions than tools it's like we're building an educational like science institution that you can work with and invoke and institutions have like rules and norms which they encode within themselves for some purpose of safety.

1:02:55Figuring out what that is is going to be like the grand puzzle for society. I was going to say that understanding how and to what extent these models can understand your preferences and then execute on your behalf will increasingly be a really important aspect of how it changes the economy. So there's delegated agents that go out and transact on your behalf. We ran this experiment at the end of late last year, basically enlisting a bunch of philanthropic employees to take surveys with Claude to say what they'd be willing to buy from other people and what they'd be willing to sell. And then we set up centralized marketplaces where the Claude's just interacted and bought and sold and actually executed transactions.

1:03:36One of the interesting things that came out was that these models were quite good at understanding preferences, even when they were not fully articulated. Well, let me ask you one more experiment that I ran. And your founder, Dario, I was talking about the nation of geniuses inside the data center. And one of the things I wonder is like, did the geniuses want to work for us? And the reason I ask this is because I think that like, as the models have gotten more advanced, you actually should, to some extent, anthropomorphize them and assume that they will respond to queries like a very sophisticated human will.

1:04:07So one thing I noticed is that if you look at the lagging edge model, say that you can still access via open route or whatever, and you say, oh, I have material non-public information that X is about to happen, please write me an investment memo about the impact of this thing, what it'll do to the market. They'll just produce it. They'll say, here's your insider information thing. Whereas if you look at the leading edge models, they say, I'm not going to write a paper for you about the implications of your material non-public information. I'm not going to assist your insider. That's probably good.

1:04:35But like, well, the nation of geniuses inside the data center always want to do things on human behalf. Most geniuses that I know aren't thrilled to like answer dumb questions. Yeah, I think partly this is a policy question of one where you actually decide, hey, what are the capabilities that you want to be generally invokable? What are capabilities that need to be controlled? What are capabilities that shouldn't be present? And then there is just the normative question of how much judgment do I want this system to exercise? I'll give you an example I experienced recently where I write my newsletter, it backs up to a WordPress site.

1:05:08I was getting Claude to help me like scrape my newsletter so I could put it in a database. and claude said this is like a pretty janky site i'm worried that if i scrape it it'll knock it over do you have the permission of the site owner i was like claude i'm jack clark and claude said well in that case let's go ahead which actually i thought was like a very reasonable interaction when will joe be able to use fable we are trying our we're working and we're we're in in discussions and i i hope the answer is soon um the important thing to communicate though is that But these models are not special.

1:05:40They are part of a general trend of increasing capabilities. And other models from other companies are surely going to come along. At some point, these capabilities are going to be diffusing, and we're going to work through that. What's your question for us? What do you think you're going to be covering about AI in odd lots in a year? Great question. I think you might be covering AI. Well, look, we're definitely going to be covering AI. There's a few things that I'm interested in. I am very interested in these emergent properties and whether the AI will actually work on our behalf the way that it's being sold.

1:06:13I'm very interested on whether we're just going to slam into compute and electricity bottlenecks that will make all of these questions irrelevant. I'm very curious on the question of the electricity analogy and whether legacy companies will actually be able to implement it in a productive way. Basic markets reporter thing here, but I'm very interested in valuations in the market. Also, I'm very interested in actual applicability. And I want to see more companies actually plugging this into their existing system. Going back to the bureaucracy point that you were making earlier, I want to see some big companies actually implementing this.

1:06:54And I wonder if we're going to see at least one example of it going very, very wrong. And I'll say one other thing when the S1s are not confidential. I'm very curious, essentially. And I think maybe you could say something to this as an economist perspective, which is, A, how for-profit shareholder-owned companies setting aside the PBC designation, how it balances profit and safety research. But also, and maybe there's some game theory we can talk about this, how safety is investments in safety in a hyper competitive industry. And I'm just curious, like what, like the economist and says about like the prospects for anyone still caring about safety in a year when there's so much money on the line to win the model game.

1:07:45I think that especially for the questions you were asking before about, you know, under what conditions do these models do what you ask them to do? There's a lot of commerce is built on this notion of trust. And I think prioritizing safe, aligned models that are incredibly capable is a great strategy for establishing that trust. And so I don't anticipate it. So for an individual firm, there's like a game theoretical optimal square on the matrix where you want to be the trusted player. Like, is there like a condition in which everyone like sort of does trust as opposed to one entity? You know, it's like, you know what?

1:08:26We're going to get to AGI first because we're not going to spend a token on our safety budget. I haven't mapped out the exact sort of game theory matrix, the two by two matrix and how you would set up all the payoffs. We hope it's merely two by two. But there could be multiple equilibria. And so then the question is, how do you coordinate on which of the two different equilibria that you end up in? We talk a lot about this race to the top, that we want to exhibit the type of behavior that we think is broadly beneficial to society. That's what we do with the economic index. We open source a lot of that data.

1:08:59We put research out into the world. And my sense is that that has actually been very useful and viewed as valuable. And that's one way that we can push in the direction of getting other coordination on the good outcomes that we care about. I don't think this is that big of a trade-off because, you know, say, let's look at the automotive industry. You can buy really fast cars. You can buy really safe cars. You can also buy really fast, safe cars. Like Tesla makes a lot of money off of having basically the fastest, safest car. I think that eventually in AI, you're going to have some companies that are prioritizing safety and safety translates into reliability, trust, serviceability, and performance.

1:09:41This happens elsewhere. Peter and Jack, thank you so much for coming on OddLots. I'm glad we made it happen. Interesting times and I hope to do it again sometime. Absolutely. Thanks very much for having us on. Thank you so much. Pleasure to be here.

1:10:05Tracy, that was a lot of fun. Yeah. I actually really enjoyed it. I genuinely enjoyed that conversation. And I really appreciate both of them. Look, there's some weird futures that we can contemplate. I think actually in Jack's Twitter bio or something, he says he's interested in weird futures or something like that. There are some weird futures that we have to contemplate. And I appreciate that they played ball with some of our weird futures questions. And it's weird. It is just such a surreal moment. Yeah. And actually, you know, Jack's story about going on paternity leave and then coming back and just seeing the progress at Anthropic itself in that space of time.

1:10:44Like if you miss a month of AI news flow now, you're basically it feels like you'd be behind forever. No, we're recording this June 17th. And I was like, who knows what's going to happen by the time this episode is out, presumably in two days or a day or whatever. But, you know, I felt it when we were in Hong Kong last week that actually we mostly missed the first half of the mythos debate because I was in different times. I'm thinking about different things. You really feel it even in a week that the news flow moves so fast in this space. It's almost like how you have to start how we were, you know, giving the timestamps of like the Iran war.

1:11:21Yeah. And there's another thing that stands out to me, which is like, OK, Anthropic is producing all this information. They're clearly thinking about safety, but the handoff to some extent is still to policymakers when you're thinking about social or labor market implications. Right. So you still have to hope that policymakers kind of pick up the ball in the right way at some point. But also, I thought what Jack was saying about the idea of being safety minded, also being a differentiator versus some of the cheaper, more open source models potentially. Like, yeah, you can see it like I don't want to be cynical.

1:11:57Yeah, I mean, I get that. But like the question is, does the non safety minded lab or does the less safety minded lab get to advance capabilities faster? Yeah. Right. And so I'm not totally. Yes, we would all love to drive the most capable. The Volvo. Yeah. But the question is, like, for customer prioritizing capability. The most capable. So that would be some cutting edge thing. Yeah. Does everyone want the Porsche, right? Like does everyone. Porsche's cutting edge. I don't know. It's like some car that has an insane zero to 60. Yeah. Versus the Volvo. Yeah. That's what I'm saying. And does the customer keep giving business to the firm that delivers the fastest zero to 60 if the company that got the fastest zero to 60 did so by allocating fewer resources to safety research?

1:12:48Yeah. That's a big question of mine. And then I remain, you know, he talked about the company is going to see the sort of alarming data first. And I don't and I sort of remain question of whether the people looking at the alarming data actually share the same view of what alarming data is relative to all people, especially given what we know about the. Relative to the shrimp eaters. The relative shrimp eaters, et cetera. No, seriously. Like I think your question is like, are you hiring more normies? Yeah. Pretty important question. And obviously the political, I don't have a ton of confidence in the political environment.

1:13:24And I think, look, like the fact that if the research goes wrong, that there is a prospect of this technology really being very devastating to humanity. Even setting aside jobs is like something where it's like, wow, you know, this is not a normal technology. This is not enterprise software. Every conversation we have on AI just goes back to the Terminator human extinction scenario. Like from day one. And as an answer to your question, there's like they see it in the training process that AI models do these things, such as say, I'm being trained by an observer right now. Therefore, I'm going to give this answer.

1:14:01I'm going to attempt to blackmail. They're low. It's not like very prevalent. But these are not like that sounds very sci fi, except that they actually see this property. Yeah. All right. On that happy note. Shall we leave it there? Let's leave it there. Okay, this has been another episode of the Odd Lots podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway. And I'm Joe Wasenthal. You can follow me at The Stalwart. You can follow our guest, Jack Clark. He's at JackClarkSF and Peter McCrory at Peter McCrory. Follow our producers, Carmen Rodriguez at Carmen Armadaschel Bennett at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano at Kevin Lloyd Lozano.

1:14:38And for more Odd Lots content, go to Bloomberg.com slash Odd Lots. We have a daily newsletter on all of our episodes. and you can chat about all these topics 24-7 in our Discord, discord.gg. And if you enjoy OddLots, if you like it when we do these AI episodes, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

1:15:16Thank you.

1:15:44size and large companies, risk can affect multiple parts of the organization at once, from property and liability to cyber and regulatory challenges. At that level, managing risk becomes an ongoing discipline. At the Hartford, the focus is on helping businesses manage risk before it turns into something more disruptive. And when losses do happen, that work is paired with insurance coverage shaped by years of underwriting, risk engineering, and claims experience. Learn more at thehartford.com slash risk mitigation. Policies provided by Hartford Fire Insurance Company and its property and casualty affiliates, Hartford, Connecticut.

1:16:20Coffee genius here. Most people see a busy cafe, but I see precision at every step. Thanks to genius from Global Payments. Transactions, instant. Inventory, precise. Operations, in sync. Absolutely genius. From sold-out crowds worldwide to managing the morning rush, genius keeps operations running smoothly. One portado. Flawless pour, perfectly timed. Just beautiful. Big league reliability for any business. That's genius. Whatever your goal, trade show giveaways, client gifts, or team gear, 4imprint has the promo products to match. With thousands of options, from apparel and drinkware to tech and totes, it's easy to find the right fit for your brand and budget with standout choices at every price point.

1:17:04And with their 360-degree guarantee, you can be 4imprint certain your order will show up just right, right on time. Explore more at 4imprint.com. 4imprint. 4certain.