OpenAI’s Big Reset + A.I. in the Doctor’s Office + Talkie, a pre-1930s LLM

22:54Kevin Roose:They are not going to make it to an IPO. They are going to sputter out and maybe end up in some real hot water. And maybe Elon Musk wins this trial. And it's sort of the end of open AI as we know it. What do you make of those gloomy predictions? Yeah, I mean, look, there are some fundamentals for open AI that remain worrisome, right? They're planning to burn tens and tens of billions of dollars in cash before they achieve profitability. They still have this very ambitious infrastructure build out that is quite expensive. And so like, I'm not going to sit here and say that like all of the numbers seem to pencil out for this company.

23:30On the whole, like if I try to, you know, put myself into the shoes of their CFO and I look through all of the stories that we just talked about, I think these seem like smart things to me. You know, it kind of seems like they're starting to dot their I's and cross their T's and get this company in a shape where retail investors will be excited to invest in the stock, which, by the way, I think they will be. So, yeah, it's one of these companies where, like, it is a generationally weird enterprise, but when I look at this particular set of stories, I think they're basically doing the right thing.

23:58What do you think?

24:00Kevin Roose:Yeah, I mean, I think there's this interesting fallacy in the AI industry where it's like, there will be only one winner, right? Everything is zero-sum. If OpenAI is having a bad month, it's because, you know, Anthropic is having a good month, or Google Demind is having a good month and vice versa. Like their sort of growth comes at the expense of all the others. And I think that feeling is shared by, among others, the executives of these companies. But I just don't think it's true. Like I think that there are going to be a handful of companies that are just going to kind of rise and fall together, right?

24:34Kevin Roose:That if your models are in the sort of top tier, you are going to be fine as long as they stay in the top tier. and the sort of rising tide of AI adoption will sort of lift all boats. That's more my feeling. Well, will this rising tide lift all AI podcasts as well, do you think? I hope so. I hope so. Me too. When we come back, it's time to take your medicine. Dr. Adam Rodman is here to tell us what's going on with AI and doctors.

25:30We'll see you next time.

25:42defense platform fighting back against impersonation and manipulation. As attackers use AI to make their tactics more sophisticated, Doppel uses it to fight back. From automatically dismantling cross-channel attacks to building team resilience and more. Doppel, outpacing what's next in social engineering. Learn more at doppel.com. That's D-O-P-P-E-L dot com. So there's a lot of noise about AI, but time's too tight for more promises. So let's talk about results. At IBM, we work with our employees to integrate technology right into the systems they need. Now, a global workforce of 300 ,000 can use AI to fill their HR questions, resolving 94 % of common questions.

26:22Not noise, proof of how we can help companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Let's create smarter business, IBM. Marketers have always had to choose, build your brand or drive sales. With YouTube, You can do both. It's where the most trusted creators and powerful AI converge to create and convert demand for your brand. That's why YouTube drives higher long-term return on ad spend versus TV, paid social, or streaming. There's no more choosing between brand or results. With one platform, you get both. Learn more at g.co slash business slash YouTube.

27:05Kevin, is there a doctor in the house?

27:07Kevin Roose:There sure is, Casey. Today, we are going to have a conversation with a doctor about AI and medicine, because this is an area where there has just been a lot happening recently, and we needed someone qualified to come in and debrief us. Yeah, you know, as we've sort of looked across the landscape just over the past few months, we've seen company after company introduce their own product at the intersection of AI and medicine. There's Chat2PT Health, Chat2PT for Clinicians. Amazon has something called Health AI. Microsoft has co-pilot health. And of course, all the while, doctors are experimenting with this technology and as best as we can tell, actually getting really excited about what they're seeing.

27:47Kevin Roose:Yeah, and this has been a huge change in my recent visits to doctors, which is that I now am having this series of conversations leading up to the visit with AI systems about what is going on. And so I am coming armed with what I believe to be good information about what is going on. And that allows me to sort of have a different, more elevated conversation with the doctor. And this is not just me. Like, people are increasingly turning to chatbots for medical information. According to some recent data, approximately a third of Americans report turning to AI for healthcare information. And companies are racing to respond to that demand by making better tools that are specifically designed for use in healthcare.

28:28Kevin Roose:So to help us make sense of the landscape for AI and medicine and healthcare, we invited back to the show one of our favorite doctors, Adam Rodman. He is an internal medicine physician at Beth Israel Deaconess Medical Center and an assistant professor at Harvard Medical School. Yeah, we last talked to him in November of 2024. And since then, he's continued to study the way that people and AI interact in the healthcare space. And we have a lot of questions for him. Like, what should we do about your rash, Kevin? Yes. So let's fork over our co-pays and bring in Dr. Adam Rodman.

29:08Dr. Adam Rodman, welcome back to Hard Fork. Oh, it is a pleasure to be here. Am I a friend of the show at this point? Well, let's see how this interview goes.

29:16Kevin Roose:You're at least a doctor of the show. You are a primary care physician. So when we last talked to you in late 2024, I think this was a moment where the medical community was starting to say, wait a minute, these AI models are getting pretty good at things like diagnostics. But I think a lot of the field was still kind of in wait-and-see mode. Now, almost two years later, we have a lot of new tools and a lot of new studies about the use of AI in medicine. So just catch us up on, like, what has been going on with AI in medicine for the last, call it, year and a half. Yeah, it's been crazy. AI in medicine has gone from, well, depending on how you measure it, it's probably the fastest adopted medical technology of all time.

30:00we went from this being super novel, almost no one used AI tools, to this being a routine part of most doctors' weekly practice. And give us a sense of, like, the AI stack for a doctor. What are the tools that they are using right now and how? And particularly, like, what are the mainstream doctors, like the people that, you know, aren't yet on the bleeding edge? The normies? Yeah, if you will. Yeah, so the biggest sort of normal doctor technology, which most of your listeners or a good portion of your listeners have encountered are what are called AI scribes. That's a sort of voice-to-text algorithm that listens to you talk to your patients and then writes a first draft of your note.

30:38And these have gone from like kind of a novel experimental technology to commodity in probably less than two years. They're everywhere. Doctors really like them and then patients really like them because they spend more time talking. And then the second sort of, I'd say, normal doctor use case is for decision support. So there's this one company called Open Evidence that has created a free tool that has gone from, again, zero to crazy numbers of adoption. I will tell you, younger doctors like my residents use it all the time. I don't know the actual numbers, but it's probably close to half of U.S.

31:17doctors are using this right now.

31:18Kevin Roose:Wow. So yeah, the statistics that I've seen are that more than 40 % of doctors now are using this, which is pretty crazy uptake for something that was just started a couple of years ago, back in 2022. In March, Open Evidence reported that in a single 24-hour period, doctors consulted the AI system a million times. I've been fascinated by Open Evidence. I've never used it myself, but I have friends who are doctors or nurses, and they have said what you've said, that basically just everyone, especially on the younger end of medicine, is just using this thing constantly. So give us a sense of how this open evidence tool works.

31:56Kevin Roose:What situations is it used for? And what are its strengths and weaknesses? Oh, that's a great question. So how open evidence works, all of these tools is a trade secret, but it uses some sort of retrieval augmented generation and an evidence retrieval tool. And they have all these deals with the big medical journal. So New England Journal of Medicine, JAMA. And when you ask a clinical query, it searches the evidence and then tries to identify high-quality sources. And then it always grounds what's coming back in the literature. So you have gray hairs like me who kind of use open evidence the way that I would use a Google search or one of the old tools.

32:34So I use it as a souped-up way to search the literature rapidly and often go to the primary sources. Or I use it as a faster way to get a reference. So a drug that I haven't dosed in a long time, open evidence pulls the drug monographs from the FDA. I can very quickly pull that up. Younger doctors, I have noticed, and I don't know this empirically, but younger doctors are more likely to ask questions like, what could be going on? Can you give me a second opinion? What is the next thing that I should do? So ways that I don't traditionally use decision support or reference tools, but sort of a new way.

33:09And of course, younger doctors also use it in the reference ways that I do.

33:14Kevin Roose:Now, are they actually uploading patient data to this? Like, or are they just sort of describing patients in generic and anonymized ways to get back some decision support? My understanding is largely number two. I'm sure the company has a good sense on how many people. I hope no one is copying protected health information and putting into it. certainly what I've observed from like my colleagues and my students, most people use it the way you would use like a search tool when you have a question, which is, hey, I like I'm giving this person ceftriaxone. What's the right dose for an intra-abdominal infection?

33:49So sort of generic questions that are being interpreted through the physician.

33:53Kevin Roose:And are there any AI tools that are integrated with patient health records? This has been an area where I think there's just been a lot of pushback of like, I don't want my personal health data, my protected health data going into one of these cloud-based AI systems. But are there hospital systems or medical systems that are bringing this stuff directly into contact with patient data? Oh, 100 percent, yes. Right now, most of the sort of in contact with patient data are less about physician-facing decisions and more about like billing. There are companies that are integrating with the electronic health record.

34:30Those are not standard yet. And then the EHR companies themselves, so Epic is obviously the biggest EHR vendor in the US. They're doing a lot of work on building in native things. So for example, at my health system, if I want to send a message to the patient, the helpful AI at the top already has like, maybe you should say this. It's usually not that helpful and I don't think I've used it once in my life. But there are a lot of those things that are being experimented on actually built into the patient's health data. I'm curious how doctors are feeling about all of this. We saw a survey from the American Medical Association that found that more than 80 % of physicians now report using AI professionally.

35:09Is that physicians racing out and grabbing these tools and bringing them to the office because they're so helpful? Or is this the classic case of a CEO saying, hey, you got to use AI or you're out of here? So doctors are BYO AI. A lot of that AI use is AI scribes and a decision support software. And I'll tell you, some people are just using straight up like ChatGPT or Gemini or Claude for the decision support software. So I think one of the reasons doctors thus far have been more positive about it than perhaps the overall population is they're largely tools that doctors are bringing themselves that they think make their lives better.

35:44And at least not yet many things that are being imposed upon us.

35:48Kevin Roose:Yeah. I've noticed that when I and my friends go to the doctor now, we often are presenting our information to a chatbot first and then coming into the doctor with sort of a readout of what the chatbot has told us. This is, of course, not a new phenomenon. People have been doing this with WebMD results for many years. But is this something that you're seeing now is that many more patients are coming to you having already discussed whatever's going on for them with a chatbot? Yes. This is the other big changes is that there's, you know, there's someone else in the exam room with me. And often it's ChatGPT.

36:21They're talking, sometimes with my hospitalized inpatients, they're talking to ChatGPT while I'm in the room with them. And I think this is, it's interesting because this is kind of a new competency for doctors. We have to talk to our patients about AI. And I have started to talk to my patients about what I think are like safe uses, what are like safe uses while telling me, and then things that they definitely shouldn't do. So my patients may talk to me more about it because I am like a doctor and an AI researcher, but like a lot of my patients are using AI routinely. Well, give us a flavor of what you're telling them because, you know, I am definitely somebody who has looked up my symptoms before I've gone to the doctor.

36:57And I would say I found it enormously helpful. But I can also imagine, you know, more skeptical doctors being annoyed, you know, at a patient telling them, you know, what ChachiBT says to do. Yeah. So here's my – I'll give you my spiel. This is, I give them a, what is it? A green light, yellow light, red light. So the green light uses are general health questions. So I have recently diagnosed with diabetes. I really love seafood. Can you help come up with a diabetic diet for me? The green light uses are also preparing for clinic visits. So I'm about to go see Dr. Rodman. I want to make sure that I ask the right questions.

37:34Here is the last note or the last thing he wrote. Obviously strip out anything identifying. don't put your personal health information and like help come up with a good questions to ask him. And then other green light activities might be like wearable data. I don't know how good they are wearable data, but I will tell you if a patient is going to give me like five years of their Apple Watch data, they're probably going to get a better from ChatGPT than from me pretending to look at five years of Apple Watch data because it's a 20 minute visit. The yellow light, Casey, I think is a lot of the things that you're saying.

38:05So I tell my patients it's okay to explore new symptoms. It's even okay to seek out second opinions when talking to a chatbot that can really help prepare you, as long as you understand that it's not a replacement for a doctor, and that is the first step to talking to a human being. So LLMs are really powerful, and I mean, there is some evidence, of course, that when any human uses them, you don't always get laboratory-level performance. They can give you dangerous advice, But diagnosis and like exploring symptoms, as long as you use it in a way to prep to see your doctor can be very helpful. The red light, what I tell them never to do is like ask medical management decisions.

38:46Like don't say my doctor said to do this. Is this right? Like I have cancer. God forbid you have cancer. Is this the right chemotherapy option? Like a lot of those decisions are so nuanced, taken so much information. Those are things that the models don't do well. And they're so sycophantic, they can convince you that they're saying the right thing even when they're wrong.

39:04Kevin Roose:Yeah. I'm curious, Adam, out here in San Francisco, there are all these fitness people and health maxers, people who love to track themselves using all manner of devices. and people are getting these full-body workups from companies like Function Health that are, you know, sort of concierge medicine things and they'll get, you know, 100 labs done and then they'll upload all that data into Claude or ChatGPT and just sort of treat it as a sort of first-line medical professional in their lives. Do you think that is a good practice or is that just making people, you know, way too worried about things that maybe they don't need to be worried about?

39:42Yeah, so that's making people way too worried about things they don't need to worry about. And this is ChatGPT. LLMs in general, I mean, the dark side of talking to an LLM about your symptoms is they are so sycophantic they can drive you into like the cyberchondria worry hole. The evidence is not there yet that the sort of large routine testing functional medicine and putting it into an LLM does anything to improve health outcomes. Now, if your LLM is telling you to work out and eat healthier, that's probably pretty good. Sleep.

40:12Kevin Roose:Yeah. What about the integrations like ChatGPT Health, which lets you sort of convert your Apple Watch or Fitbit data into something that ChatGPT can analyze? There's also a new version of ChatGPT for clinical use called ChatGPT for clinicians. Are any of these integrations or projects more promising in your view? Not yet, but I think it could be at some point. I mean, so ChatGPT for health pulls in your data from the medical record. and lets you chat with your medical records. Now, reason number one for concern is privacy. That's obviously going to have your entire medical history going to an AI company.

40:52It's also going to not be redacted by you in a way to remove identifiable things. Reason number two, I think if we're talking about health record data, it's really messy. They include tabular data. They include copy forwarded data that's been copied and pasted. And they also, if you've ever read your health records, they include things that are wrong. There's a lot of errors or misdocumented things in your health data. And it turns out that just copying a bunch of information, like, LLMs aren't magical. You can't just copy your entire medical record in and think that you're going to get good performance.

41:26And I would never bet against the technology. I think that we will get to the point that we have ways to build representations of humans and understand their health. But right now, there's like no advantage to just dumping everything in an LLM, which is what ChatGP2 for Health theoretically would allow you to do in a way that would allow you to better understand your health. I'm curious if you saw this trial they're doing in Utah where you can use an AI agent to autonomously renew prescriptions for almost 200 routine drugs. Yes. There's apparently some human review, but mostly this is automated.

42:00Is that a good idea, a bad idea? Well, so globally, no, we should not be having LLMs write prescriptions for people. The trial in Utah in particular is a refill. So a doctor has already written a prescription within the last 12 months. And I guess the idea is that it saves the primary care doctor time from having to review and refill. I'll tell you, if you talk to most doctors, yes, it is annoying to get refill requests. No, that is not the thing that drives us crazy. This is not like a use case that we're screaming for. I think it's being done as a proof of concept of can this work in the real world.

42:36This trial in and of itself is not dangerous. Prescription refills, and I think there's no opiates. There's no dangerous drugs in it. And a doctor has to have written the original one. But even if it does work in this, that does not mean we should be having autonomous AI systems write new prescriptions. That is not safe, and it's not a good idea yet.

42:53Kevin Roose:See, I think this is a case where like this is sort of rent seeking behavior on the part of doctors or doctor organizations. Like when I have gotten refills for prescriptions, I meet with a doctor for, you know, six to eight minutes. They say, how's it working? I say, great. They say, are you having any side effects? I say, no. they say, okay, I'll write you a refill. And the whole process just seems totally designed to like get me to pay up for another doctor visit and not give me any actual good medical advice. So if I can play devil's advocate, like, do you think that there, that the sort of resistance to programs like this are motivated by just wanting to keep people coming to the doctor and paying for those visits?

43:42So first, aren't most of your prescription refills just done as in you, you call the pharmacy and they send an automated thing to your doctor and they click the yes button and you never talk to them?

43:49Kevin Roose:No, for some, they make you actually do an office visit and maybe they want you to take your blood pressure again or whatever. So I'll do the devil's advocate back. Let's say I prescribe a fairly common antidepressant and they want it to be refilled. What I don't know is that this patient may be, the silly question you get in the clinic, may be new lesions forming in your mouth, and it's an early ulcer. And if we don't pick it up within 24 to 48 hours, you may develop like Steven Johnson syndrome, so potentially life-threatening complication. And the reason there are certain types of drugs, including antihypertensives, is that they can be high risk and we need follow-up.

44:28Now, is that everything? No. And definitely there should be more things over the counter. I don't think that most doctors are sitting around saying, I wish I had more medication follow-up visits. And the reason some of these things exist is that there can be very dangerous symptoms. Yeah, so keep going to the doctor, Kevin. We can't have you developing those lesions. It's too important to the show. Let me ask you about another one. This one actually seemed like just an unqualified good. The Mayo Clinic announced this week Red Mod, this AI system that identified subtle changes in routine CT scans up to three years before a pancreatic cancer diagnosis.

45:04And this was like many, many, many percentage points better in detecting pancreatic cancer than human beings. So to me, this is like the sort of thing I keep waiting for AI to do, and it seems like it's actually doing it. And of course, that's very exciting with something like pancreatic cancer, which is notoriously difficult to detect and has like very low survival rates. Yeah, and this is so like completely out of the discourse of like autonomous AI agents, there's really exciting stuff happening. So the Mayo Clinic, there've been some great studies on breast cancer detection. A lot of these algorithms have gotten so good that they're able to identify breast cancer better than, I shouldn't say better than people, but in a workflow that has a good detection rate.

45:43And then in picking up like potentially cancerous polyps when you get a colonoscopy. So there's a lot of exciting and really positive things that are coming. And I mean, at the end of the day, we'll need to see how RedMod works in the real world and a trial. But I'm really optimistic about that sort of technology.

45:59Kevin Roose:Do you think that if AI meaningfully extends life expectancy for people, it will be because of new AI discovered drugs or because of changes to routine health care that are made more efficient or more accurate by AI? Number two, I think that when you talk about AI drug discovery, the part of the pipeline that's so difficult is not necessarily the coming up with the new compounds. It's running the clinical trials and getting it through the regulatory process, which can probably be sped up, but not as much as the discovery. You know, if we get this right, there's so many people in the U.S. who don't have access to a doctor, who don't have access to very basic medications, who can't control their diabetes because of lack of access.

46:45And I'm really hopeful that if we do this wisely, we can, you know, get people more access to care, which I'm doing my knock on wood, hopefully will improve health outcomes. So like all of this, I think the potential benefits are like less exciting. They're getting people, more people the bread and butter and getting, you know, more people to have less heart attacks, more people to have less strokes, more people to get their cancer screening and not necessarily like, oh, we cure aging with some sort of new AI-discovered CRISPR technology. Are you at all surprised, though, that we haven't yet seen the first, like, AI-discovered wonder drug?

47:19I mean, the biggest wonder drug of my career has been the GLP-1s, which was... I started using it when I was a resident, so we had it for a really long time, and we had to, like, repurpose a drug for diabetes. So, no, I'm not. Like, medicine and science is just kind of messy, and it's... There are always those stories about, like, you know, we discover something amazing and makes it like penicillin. But even penicillin took like 20 years to get into human beings. So no, I think we will see AI discover drugs. I think it's just the benefits from AI are going to be like the benefits from medicine.

47:53It'll be a lot less exciting than people think, but still important.

47:57Kevin Roose:So there's a lot of worry right now about sort of AI in schools and education, some of the cognitive atrophy that people are worried about. oh, if we start using AI to do all of our work, we're not going to have the basic skills. Is that something you're worried about for like recent medical school graduates where maybe they would have had to hold all this stuff in their brains a few years ago and now they can just ask a chatbot and maybe that's going to erode some of their skills as a physician? Yes. So that is the biggest worry that I actually have about sort of the short to medium term is de-skilling of the workforce.

48:33We have some evidence. There was a sort of scary study last year from Poland on a trial where they gave doctors, not a language model, but a polyp detecting technology. And they looked at their ability to detect polyps, so potentially cancerous lesions in the colon before using it, and then after using it for three months. And when not using it, their ability to detect polyps dropped by six percentage points. So these are skilled doctors using technology, and they lose six absolute percentage points of their ability to detect potentially cancer in three months. and then imagine that you're learning to do it for the first time.

49:06Will you ever gain those skills? So like at Harvard Medical School, and medical schools I think everywhere, this is our big worry, which is how will this affect us to train the new generation of doctors? And it's like every other field. Like you talk about debugging code. In order to become a new doctor, you go through all this training because you need to make mistakes and you need to have someone above you who knows what's going on so those mistakes won't hurt patients. And that's just how education works, and this threatens that. I mean, it's interesting, though, because it's like, you know, it's probably true that because I had access to a graphing calculator, like if you took it away from me, I'd be worse at like plotting parabolas on a graph.

49:45But the solution to that is that I just keep using the calculator, you know? So, like, I'm not sure how big of a problem this really is. I'll also say there's something there's something deeply ingrained in human society that middle aged people complain about young people. So I think whenever we talk about de-skilling, we have to keep that in mind. Yeah.

50:03Kevin Roose:I mean, for what it's worth, like, I want my doctors to be using AI models. I want them to be consulting the hive mind before they weigh in on my specific condition. It doesn't threaten me as a patient to know that they are using open evidence or something similar. But I'm guessing for a lot of people, that would seem strange. And maybe there are some physicians who don't advertise how much they're using AI because their patients might think less of them. Do you think that's happening? Oh, yeah. Yeah, I think there, in certain situations, in certain places, I bet there's social pressure to say that you're not using AI, that there's some ego on the line.

50:40I don't see that. But again, I'm an AI researcher, so I don't think anyone would say that to me. To me, it just seems weird to, like, to hold as your standard for what makes a good doctor that they have memorized, like, a maximum amount of material. Like, that's basically what we're talking about. It's sort of like, you know, the taxi drivers in London that have to, like, learn every single street and, like, hold them all in their heads. It's very impressive, but I'm fine with them using the GPS. Yeah, and I think it's less about, so it is about memorizing. It's more at this point, right, with where AI is now.

51:09It's more having sort of that knowledge and we'll call it wisdom to know when the system might be suggesting something wrong, which is something that right now, and this may change, we get by seeing a lot of cases and reflecting on them. So right now, you're going to get the best performance if you have an experienced human trained in the old-fashioned way with an AI system. But I think your guys' point is at some point that might not matter, right? The AI systems might just outperform all of us. And then, yeah, I guess it's like just use the graphing calculator. But we're not there yet.

51:38Kevin Roose:Would the AI models be better if we were less protective of privacy for medical data? I mean, that's such a loaded question. So the first thing that I'm going to say before I answer that is patient privacy is very important. And we should respect people's privacy and their ownership of their data. But yeah, so in short, like the reason they're not better at certain things is that you need to get LLMs better. You need to label and then train them on the sort of labeled health data. And in the U.S., there are appropriately many restrictions on how health data can be used. I suspect that these companies like OpenAI, by having ChatGPT for health, they will gain some more of their own data, which they say they're not going to train on.

52:21I trust that they're not going to train on it, but they'll be able to use that data to at least evaluate their models and try to make them better.

52:28Kevin Roose:I think they should train on it. I mean, obviously, that'd be a huge illegal violation of privacy, but it would also make the AI doctors better. Much better. And I think, you know, a lot of people would be sort of willing to make that tradeoff. So I at least think there should be a little checkbox when you go to the doctor that says, like, I'm OK having my personal health data used to train AI models. I for one would check it. Yeah, in exchange for like 30 % off your giant medical bill. You get a coupon. You get a coupon, you know. You get like your next Ozempic shot is 20 % off. Is on the house.

52:59Kevin Roose:Exactly. Well, that's a good place to leave it. Dr. Adam Ronman, thanks so much for coming back and keep us posted on what is going on in medicine. My pleasure, guys. Thank you very much. Thanks, Doc.

53:14Kevin Roose:When we come back, we'll talk about Taki, an LLM trained only on data from before 1930.

53:46native. What if AI handled the busy work for you? Updating JIRA tickets, drafting campaign briefs in Confluence, writing project updates, all without you lifting a finger. Meet Rovo, AI that works where your team already works, using your company's data and permissions to take action. Get started with Rovo at rovo.com.

54:09Kevin Roose:I'm Paul Tenorio. I cover soccer for The Athletic. And I'm Amy Lawrence. I cover football for The Athletic. Whatever you call it, the biggest competition in the sport is happening right now. And The Athletic's World Cup coverage has everything you need to follow the tournament. There's 48 countries taking part from the tiny island of Curacao to the five-time champions Brazil. Even if you don't know your offside from your onside, if you're eager to know more about the teams, the matches, all the stories on and off the pitch, we've got you sorted. Maybe you're the kind of person who's already up early every weekend, waking the neighbors when your favorite club scores.

54:44Kevin Roose:We'll make sure you get equipped with more information, more insight than anyone you know. We've got more than 70 obsessive reporters on the ground, covering the ins and outs from every game. I almost forgot to mention the best part, Amy. Free access to the Athletics World Cup coverage in our app. Download the Athletic app and see you there.

55:06Kevin Roose:Well, Casey, usually on this show, we are talking about the future, but today we are going to take a trip back to the past, specifically to the year 1930. What was happening in the year 1930? Oh, my goodness. Well, of course, we were in the middle of the Great Depression. My grandmother had recently turned 11 and was looking forward to getting her first store-bought dress in just a few years. So this is a new language model, a vintage LLM called Talkie, and it is trained exclusively on data from before 1931. This is a research project built by three guys, David Duveno, Nick Levine, and Alec Radford, the lead author of the GPT-1 paper, a former OpenAI researcher over there.

55:56Kevin Roose:And this is a fascinating project that has been burning up my timeline this week because this is an experiment in what happens if you only feed a large language model data from before a certain cutoff. Yeah, and obviously there are a lot of, you know, kind of character-based chatbots on the internet that will give you the experience of talking to somebody from the past. But what makes this project different is that they try to limit themselves to training data from that time and before. The hope was that it would avoid any kind of contamination from what came after. And as you'll hear, they have some really interesting and potentially useful ideas about what this kind of LLM might one day be used for.

56:36Casey, have you spent any time playing around with this model? I have. I tried to ask it the most 1930s question I could think of, which was, say, what's the big idea? What did it say? It said, the big idea is to popularize. And I said, popularize what, fella? And it said, popularize a sport. And I said, I'm gay. So that's kind of where we let that one drop. And it said, gay, you're happy? Yes, exactly. It said, your heart must be light, sir.

57:07Kevin Roose:Yeah, I love this experiment. I love like weird niche language models. One of my favorite language models of all time was Golden Gate Claude, which was the special version of Claude that was like pathologically obsessed with the Golden Gate Bridge. I would put Taki in sort of that category of like an experimental research model that is maybe not all that useful on its own, but like helps illuminate something interesting and important about these language models and what happens when you train them in specific ways. So today we wanted to talk with one of the creators of Taki. We are going to bring in David Duveneau.

57:42Kevin Roose:David is an associate professor at the University of Toronto who researches AGI governance and catastrophic risk mitigation. And he is one of the co-creators of Taki. And there's really Duvino a better person to talk to about it. That's true.

58:03Kevin Roose:David Duvino, welcome to Hard Fork. Thank you very much, Kevin. So this project, Taki, is fascinating. It is a vintage LLM. Explain why you and Nick and Alec made this thing. So this all started a year ago. Was me and Nick were interested in forecasting, Like specifically, can we teach machines how to forecast like five or 10 years ahead of time? Like what is the big picture going to be? Just because we have our own sort of like pet ideas about what the future is going to be. We don't think people should take our word for it. And we also don't think that people should trust machine forecasts unless they have a track record going back like decades and decades.

58:40So the idea here is that if we could build a model who really only knew about the world up to a certain date, we could ask it to forecast like five or 10 years ahead of time. Like ask it, what's the New York Times headline going to be five years from now? or is there going to be another great war or something? And we can iterate and see, like, what kinds of things are predictable, what does it take, like, how far out can things be foreseen? And then hopefully, eventually, we'll have machines that have, like, a 100-year track record of forecasting. And then we can ask them, you know, in 2026, what do you think is going to happen, like, you know, two or four, eight years from now?

59:13And we'll have an idea of how much to trust those forecasts. It's a fascinating idea, but it strikes me it requires you to have like really good data. So in this case, really good pre-1930s data, I'm going to guess that was harder to obtain than just, you know, going out and like crawling Reddit or, you know, and everything else that the frontier models have access to. So how did you face that challenge and where'd you get this pre-1930s data? Yeah, so I should mention, you know, there's a ton of groups doing a ton of awesome archival work here. The first data set we got excited about was Institutional Books, which was Harvard Library, scanned like 1 % of their entire collection.

59:51And so they had tons of data from like 1800s, early 1900s. Like there's a whole bunch of different groups doing tons of work. It would take a long time to enumerate this, but, and also like OCR has just gotten a ton better just in the last like six months even. And so there's always been lots of projects to like automatically digitally scan this data, but it just hasn't been very high quality until very recently.

1:00:14Kevin Roose:And I assume that part of the reason you chose the cutoff date of around 1930 is because that's when sort of works become public domain. Anything after that is copyrighted. Are there any other reasons you chose that specific point in time? No, that was entirely it, is that we want to make everything publicly available in open source. And 1930s is just the sort of most recent date that has almost zero legal headaches with releasing data or anything like that. So I've been fascinated by seeing what people are trying with this model. People are having it make predictions, but also asking it about its favorite authors or its opinions of major historical figures.

1:00:58Kevin Roose:What have been the experiments that have been most interesting to you? Yeah, the fun things that I've seen people do is, I mean, a lot of people like to ask, like, what's 2026 going to be like? And the model has sort of very philosophical answers about how, like, well, we'll have figured out that war is bad. We'll have, like, a much more peaceful civilization. Or sometimes it says, like, it's the end times. I mean, it's a very inconsistent model, and it's not quite smart enough to really, like, you know, think things through in a systematic way. It kind of just gives you vibes. Now, that brings up a sort of interesting, like, wrinkle of like the kind of LLM this is, because if you were to ask like a frontier model today to predict the future, it would not only be trying to guess a statistically likely sequence of words, right?

1:01:41It would also be like doing some reasoning. Italki is not doing that, right? So that just sort of seems like by the way that it is built, we would expect it to be less good at forecasting as the models we have today. Yeah, absolutely. This is a very baby steps model. The basic fine tuning for reasoning and the scaffolds, like the super forecasting scaffolds that we know just improve anyone's reasoning. Like, you know, think of the different distinct possibilities and assign them each sub probabilities. The model's just not really smart enough to follow these kind of detailed multi-step instructions yet.

1:02:11So, again, we just wanted to release the first thing that we did, but it's like a, there's a clear path to adding all these refinements. So you do plan to add reasoning as you go?

1:02:20Kevin Roose:Oh, absolutely. Okay. People have also pointed out that the model behind Taki seems to know about some things that it probably shouldn't know about, like the rise of Hitler, the presidency of FDR, things that didn't happen until after its data cut off. Is that proof that there's been some kind of contamination of the training data with more recent data? Oh, there's definitely contamination. And this is sort of like one of the ongoing, like things that we're going to have to just keep revisiting again and again and refining. So we have a classifier that tries to look for things that are anachronistic.

1:02:51And especially if you want to use this for forecasting or to evaluate forecasting, it's really important that we really nail this issue. so we have all sorts of ideas for like canaries and things that we think uh the model should just never assign any likelihood to like think of i don't know nagasaki and hiroshima like before world war ii like those those two towns would just never show up in the same sentence ever almost except for like some weird weird coincidence um so you can just tell whether there's been um leakage about important events if the model just thinks that there's any chance that you'll see those particular names together.

1:03:23So this, anyway, like we've done, we've made a bunch of efforts to avoid leakage. We know there's leakage right now, so you shouldn't use it to evaluate your forecasting scaffold yet.

1:03:31Kevin Roose:But like, how is it getting that data if it's only being fed, scanned, OCR'd books from archival sources? Because archival sources have wrong dates in them all the time. Or it's kind of unclear what the like date of a text is because there's like an updated edition or sometimes there's like a preface that's been added later, or sometimes even just in the middle of the text, they'll be like, someone inserted some future notice, like, you know, note, like later on, like historian's note, da-da-da-da-da. And so it's just really hard to check all these little edits that people make, and then they still maintain the original publication date on the metadata.

1:04:09Kevin Roose:I see. I asked Taki what it knew about me, and it said, Kevin O 'Hara, which is not my name, was born in Dublin in 1840, and having been educated at the School of the Christian Brothers, became a teacher in it. He afterwards adopted the profession of journalism and was for some years connected with the staff of the Nation newspaper. It also said I had written several popular songs, including Molly Astor and The Irish Immigrant. Now, obviously, most of that is wrong, but it did connect me to journalism, which I found interesting and maybe like some other evidence of some data contamination. But like, is this thing accessing the internet in some way?

1:04:48Kevin Roose:Or like, how would it have known that I, or at least Kevin O 'Hara, this character sort of connected to me in the model, was a journalist? You know, that's a great question. I guess I'll say the training data was like 240 billion tokens. And it's just, there's just sort of like this vast ocean of stuff. So like maybe there was a list of journalists that got put in somewhere that had your name in it. I mean, I guess one thing about this model is it hallucinates like crazy. And, you know, this was a huge problem with the chatbots that people were meant to use professionally. And I think it's been addressed to a large extent in like frontier models.

1:05:22But we made zero effort to address that in any of our post-training so far. Kevin, would you sing a few bars of the Irish immigrant for us?

1:05:31Kevin Roose:You know, I don't want to waste his time here. All right. We'll save that for later. Speaking of problematic content, some people found that Taki gives racist responses to questions that are basically like, you know, would you let a black professor teach your child? I can see how that might be historically accurate, but I'm curious if you anticipated it and how you feel about it. Yeah, so it was also very clear to us that it had these kind of responses. I mean, I guess I'm a professor myself, and my sort of first instinct is, like, let's let people see this if they want to and just don't surprise anyone and don't be flippant about it because, you know, it really can be upsetting to some people, and especially if we just treat it sort of insusantly.

1:06:14So the way we threaded the needle was we did zero filtering of the data set for problematic content. We wanted to just show what the actual state of knowledge or state of thought was in the past. It would defeat the purpose of the project if we put our thumb on the scale. But for the public demo where you can talk to Taki, we just had a modern model with modern sensibilities just read every response. And at the end, once it's generated, if it is deemed problematic, just like slap a warning and say like, oh, this might have something upsetting. Just like click if you want to see it. Right.

1:06:50Kevin Roose:The description I loved, this came from Gavin Leach today, was that Taki is creating beautiful prose by a terrible person, which is consistent with some of my tests, which is like, this thing actually does write quite well. And actually to my ear, like much more literary than some of the more recent models trained on more recent data. But yeah, it is not, it is clearly the product of its time or at least the time of its data. Yeah, yeah. And I mean, the prose is really cool because it's like very refreshing style. And actually, if you feed it to one of the AI detectors, it usually says like 100 % human, which is kind of funny.

1:07:30But then I guess, as you mentioned, like a terrible person, I mean, right now it's kind of ends up being this sort of like average person and depending on like it'll just randomly answer all sorts of different voices um but that's one of the next things we're planning to work on is helping you talk to more specific people or in specific sort of states of knowledge or times and places because that i think allows you to answer more coherent questions than just talk to like the hive mind of 1930 or whatever speaking of the hive mind i saw another person asked talky basically the person that told it that it was from the future and would tell talky anything it wanted to know about the future.

1:08:06And Taki's first question was, how did universal peace come about? Which was like the most heartbreaking thing I think I've ever read from a large language model. Like, what does that tell us about the time period or about the training data? Well, I guess I'll say in general, futurism is a place where people don't actually often try to predict the future very hard. And they more, they kind of project their values. And if you ask someone, what do you think is going to happen? They usually fill in something about what they hope is going to happen. And I think that was also true 100 years ago. So the trick is to get Taki out of the wishful thinking mode and actually into brass tacks of what do you actually think is going to happen.

1:08:43It's just funny because I think if I could talk to an LLM from 21, 26 today, I would just sort of be like, are the humans still alive? What's going on with the climate? How many people did the robots kill? It would just sort of be a very different set of questions than Taki seemingly wanted to know.

1:09:07Kevin Roose:Now, are you going to point Taki at any big sort of scientific discoveries and see whether it can make them? I mean, Demis Asabes at Google DeepMind has this sort of theory that AGI should be able to discover Einstein's theory of relativity if you just give it all of the pre-existing scientific literature at the time. Are you hoping to use this model or a descendant of this model for anything like that? Absolutely, yeah. So one of the, like, I think Nick especially is interested in this question of, given a state of knowledge, sort of how much would it take to, like, how far ahead can you just, from pure reasoning, advance your state of conceptual understanding?

1:09:48And, like, classic examples are, like, you know, some of Einstein's discoveries, which really didn't require experiments. They just required putting the pieces together. And there's actually another project called Machinist Miriablus, that took a training cutoff of 1900 and tried to get it to see if they could rediscover special relativity. I mean, the thing is that the models that those people did those experiments on were like, I think, 3 billion parameters, just not smart enough to do very much. So he showed that you could hold his hand at a certain point and it would kind of get gesture in the right direction.

1:10:17But to do the kind of systematic reasoning and math that Einstein had to do, we probably need another, let's say, 10x in parameters at least.

1:10:27Kevin Roose:David, what are you building next? Are you going to build a bigger version of Taki and keep trying to get it to perform better? Yeah, so there's a few things that we want to do. So like obviously making the models bigger. And so, you know, right now the model is still smaller than GPT-3 was, although bigger than GPT-2. So, you know, Alec kind of says that there's a bit of a phase change around like 100 billion, 150 billion parameters where the model starts to be smart enough to actually have a back and forth conversation with. Obviously, scaling up the data set and OCR efforts. And right now, everything's just mostly English just because we can evaluate.

1:11:02We can quality check the English text because we're all native English speakers. But we want to obviously broaden this repertoire. Working on the filtering is obviously a big one. And then obviously, I mentioned all this, how do we even evaluate the forecasting ability? That's another big question. If you put the model in a robot, would that be a walkie-talkie?

1:11:22Kevin Roose:You can ignore him. Anyways, I'm going to go. Well, David, fascinating experiment. And people can go try Taki for themselves. It's at Taki-LM.com. What's a good goodbye to a podcast guest? Don't worry about what a podcast is. okay well as Taki says a pleasant journey to you sir thank you very much sir thank you David

1:12:19This podcast is supported by Atlassian Rovo, AI that takes your team from AI novice to AI native. What if AI handled the busy work for you? updating Jira tickets, drafting campaign briefs in Confluence, writing project updates, all without you lifting a finger. Meet Rovo, AI that works where your team already works, using your company's data and permissions to take action. Get started with Rovo at rovo.com. I'm Winnalou. I write the game Connections, one of the puzzles from New York Times games, and I love horror movies. I love my dog, and I love trying to trick you. I'm Tracy Bennett. I get to pick the wordle word every day, which is not as easy as it sounds.

1:13:03The fun fact about me is that I am descended from a witch who was put on trial in Salem. New York Times games are made by people, like the ones you just heard from. Go to nytimes.com slash games to start playing today.

1:13:21Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Viren Pavich. We're fact-checked by Caitlin Love. Today's show is engineered by Daniel Ramirez. Original music by Marian Lozano, Diane Wong, Rowan Nemisto, and Dan Powell. Video production by Sawyer Roque and Chris Schott. You can watch this whole episode on YouTube at youtube.com slash hardfork. Special thanks to Paula Schumann, Pui Wing Tam, and Dahlia Haddad. You can email us at hardfork at nytimes.com with your most recent diagnosis. We'll tell you if we think you should get it looked at. Thank you.