In short
Big Technology Podcast Episode Notes
Episode Title
AI’s Drawbacks: Environmental Damage, Bad Benchmarks, Outsourcing Thinking — With Emily M. Bender and Alex Hanna
Episode Description In this episode, Emily M. Bender and Alex Hanna discuss their book "The AI-Con," addressing the hidden costs of large language models, including environmental damage, labor issues, and flawed benchmarks. They engage in a robust debate about the utility of chatbots versus their limitations, and explore various concerns surrounding generative AI.
---
Key Themes and Concepts
- The AI Con
- Definition: A layered critique of AI technologies, particularly large language models (LLMs), described as more of a "parlor trick" than genuine intelligence.
- Bottom Layer: Perceived usefulness of chatbots based on a false sense of consciousness.
- Top Layer: Misleading claims about applications in sensitive sectors like law and medicine.
- Environmental Impact of AI
- Energy Consumption: Discussed the significant energy costs associated with operating AI models, including training and inference.
- Lack of Transparency: Companies often do not disclose the energy usage of their models, complicating the ability to assess true environmental costs.
- Community Effects: Instances in places like Oregon and Memphis highlight negative consequences on local resources (e.g., water shortages).
- Benchmark Gaming
- Critique of Benchmarks: Benchmarks often lack construct validity, meaning they do not effectively measure what they claim to.
- Training to the Test: Companies may develop benchmarks that favor their technology, undermining genuine progress and evaluation.
- Real-World Implications: Models may perform well on arbitrary benchmarks but fail in practical applications.
- Health Sector Applications
- AI in Medicine: Discussion around using AI for transcribing medical notes.
- Efficiency vs. Care Quality: Concerns that reliance on AI for transcription could reduce the quality of patient care, as it removes a reflective practice from doctors.
- Privacy Issues: The use of AI as ambient listeners poses significant privacy risks for patients.
- The Doomer Perspective
- Critique of AI Doomerism: Skepticism about extreme predictions of AI leading to catastrophic outcomes (e.g., "paperclip maximizers").
- Focus on Current Issues: The conversation emphasized the importance of addressing the real and immediate harms of AI rather than speculative fears.
- Real-World Impacts: Acknowledgment that the technology can indeed pose risks, but a suggestion to prioritize tangible concerns over speculative ones.
---
Key Arguments and Discussions
AI as Useful vs. AI as a Tool
- Utility of Chatbots: The guests argue that while chatbots can mimic human-like text generation, their practical utility in professions like medicine is overstated.
- Verification: Users must verify information generated by AI due to potential inaccuracies.
Implications for Employment
- Job Market Concerns: Emphasized that AI's integration into workplaces could lead to job displacement and further pressure on workers, particularly in healthcare.
- Quality of Work: The risk of reducing meaningful human interaction and professional accountability in favor of efficiency is a central concern.
---
Conclusion This episode presents a critical examination of generative AI, advocating for transparency about its environmental impact, scrutinizing the validity of performance benchmarks, and challenging the narrative that these technologies are unequivocally beneficial. Bender and Hanna highlight the need for nuanced conversations about AI's implications on society, labor, and the environment.
---
Additional Resources
- Book: *The AI-Con: How to Fight Big Tech's Hype and Create the Future We Want* by Emily M. Bender and Alex Hanna
- Related Work: Articles and research cited throughout the episode emphasize ongoing discussions about AI's role in society.
---
Feedback
- Enjoy the podcast? Consider leaving a five-star rating in your podcast app.
- Questions or feedback? Reach out to bigtechnologypodcast@gmail.com.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:00Two of AI's most vociferous critics join us for a discussion of the technology's weaknesses and liabilities and a debate on the finer points of their arguments. We'll talk about it all after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We're joined today by the authors of the AI Con. Professor Emily M. Bender is here. She's a professor of linguistics at the University of Washington. Emily, welcome. I'm glad to be here. Thank you for having us on your show. My pleasure. And we're also joined by Alex Hanna, the director of research at the Distributed AI Research Institute.
0:35Alex, welcome. Thanks for having us, Alex. Alex. Always good to have another Alex on the show. So look, we try to get the full story on AI here. And so today we're going to bring in, I think, two of the most vocal critics on the technology. They're going to state their case. And you at home can decide whether you agree or not. But it's great to have you both here. So let's start with the premise of the book. What is the AI con? Emily, do you want to begin? Sure. So the AI con is actually a nesting doll situation of cons. Right down at the bottom, you've got the fact that especially large language models are a technology that is a parlor trick.
1:12It plays on our ability to make sense of language and makes it very easy to believe there's a thinking entity inside of there. this parlor trick is enhanced by various UI decisions. There's absolutely no reason that a chatbot should be using I, me pronouns because there's no I inside of it, but they're set up to do that. So you've got that sort of base level con. But then on top of that, you've got lots of people selling technology built on chatbots to be a legal assistant, to be a diagnostic system in a medical situation, to be a personalized tutor and to displace workers, but also put a band-aid over large holes in our social safety net and social services.
1:53So it's cons from the bottom to the top. Okay. I definitely have things that I disagree with you in places on, and we will definitely get into that in the second half, especially about the usefulness of these bots and whether they should be using IRME pronouns and the whole consciousness debate. We're going to get into that. I don't think any of us think that these things are conscious. I just think we have a disagreement on how much the industry has played that up. But let's start with what we agree on. And I think that from the very beginning, Emily, you were the lead author on this very famous paper about calling the large language models stochastic parrots.
2:33And at the very beginning of that paper, there is concern about the environmental safety and the environmental issues that large language models might bring about. So on this show, we talk all the time about the size of the data centers, size of the models. And of course, there is an associated energy cost that must be paid to use these things. And so I'm curious if you, Emily, or you, Alex, you worked at Google, right? So you probably have a good sense of this. Can you both share, like quantify how much energy is being used to run these models? So part of the problem is that even, you know, even if you're working at Google, you are directly working on this, they're not very public estimates of how much cost there is.
3:22I mean, the costs vary quite widely. And the only cost I think that we know was an estimate being made by folks at Hugging Face that worked on the Bloom's model, because they were able to actually have some kind of insight into the energy consumption of these models. So part of the problem is the transparency of companies on this. You know, as a response at Google after the Stochastic Parents paper was published, one of the complaints from people like Jeff Dean, the SVP of research at Google, and David Patterson, who's the lead author of Google's kind of rebuttal to that, was that, you know, well, you didn't factor in XYZ, you didn't factor in renewables that only we talk about at this one data center in Iowa.
4:12We didn't factor into off-peak training. And so it's part of the problem. I mean, we could try to put numbers on it, but there's so much guardedness about what's actually happening here. We can't quantify it. We don't know when it comes to model training. We might have something like we know the number of parameters that are in a new model or in an open weights model like LAMA, but we don't know how many kind of fits and starts there were with stopping training and restarting or experimenting. So, you know, we could speculate, but we know it's a lot because there are real effects in the world right now.
4:51What are those effects? What are those effects? So you see communities losing access to water sources. You see communities, you see electrical rids becoming less stable. And this is starting to be, I think, very well documented. There's a lot of journalists who are on the beat doing a lot of good work. And I also want to shout out the work of Dr. Sasha Luccioni, who's been looking at this from an academic perspective. And one of the points that she brings in is that it's not just the training of the models, but of course, also the use. And especially if you're looking at the use of chatbots in search, instead of getting back a set of links, which may well have been cached, if you're getting back an AI overview, which happens non-consensually when you try Google searches these days, right?
5:33Each of those tokens has to be calculated individually. And so it's coming out one word at a time, and that is far more expensive. I think her number is somewhere between 30 and 60 times more expensive, just in terms of the compute, which then scales up for electricity, carbon, and water than an old-fashioned search. I would also say that speaking about existing effects. There's also a lot of reporting coming out of Memphis right now, especially around the methane generators that XAI has been using to power a particular supercomputer there called Colossus there, specifically around emissions there.
6:17Affecting Southwest Memphis, traditionally a black and impoverished community. There's also reporting on, well, actually in research from UC Irvine in which looking at backup generators and emissions from diesel that are connected to the grid, but just because the SLAs on data centers are incredibly high, you effectively need some kind of a backup to kick in at some time, and that's going to contribute to air pollution. And which communities have been affected by the loss of water due to AI data centers? So I think the best reported one is the Dulles in Oregon. I mean, I think that's the one that is the best known.
7:05That is kind of pre-AI in which we're focusing on the development of Google's hyperscaling. And it wasn't until the Oregonians sued the city that we knew that half of the water consumption in the city was going to Google's data center. That was before generative AI. That was before generative AI. I mean, we have to imagine the problem is probably exacerbated right now. But do we know that? I mean, you both wrote the book on this. So we have we certainly point to environmental impacts as a really important factor. It is not the main focus of the book. I would refer people to reporting of people like Paris Marks over at Tech Won't Save Us.
7:48did a wonderful series called Data Vampires, looking at, I think there was stories in Spain and in Chile. And yeah, so this is, you know, we are looking at the overall con and the environmental impacts come in because it is something we should always be thinking about. And also because it is very hidden, right? When you access these technologies, you're probably sitting, you know, looking at them through your mobile device or through your computer and the compute and its environmental footprint and the noise and everything else is hidden from you in the immateriality of the cloud. I would also say that, I mean, the reporting on Memphis, I want to give a shout out to the reporting in Prism by Ray Levy Uyeda.
8:32I don't know if I'm pronouncing their surname correctly, but they have an extensive amount about the kind of water consumption of this saying that this would take about, I think, a million gallons. I'm checking it, but I'm looking at the reporting on it. I think I'm seeing the exact number on this. I'm going to look at it. Yeah, so they're focusing a million gallons of water a day to cool computers. They don't. They're saying that they need to build a gray water facility. to do it. I mean, this is not anything that any, that these facilities don't exist yet, so they'd have to be built. But I mean, this thing is already being constructed and is using water.
9:22So I mean, I don't think it's a far cry to say that this is happening in an era that was in the hyperscaling era, in pre the Gentry of AI era. I mean, it's the unfortunate fact about it is that a lot of these community groups are fighting this on a very local level. And a lot of these things are getting underreported on just because, but from what we know from the fights in the Dulles and in London County and, and parts of rural Texas, I mean, we would be surprised if this similar kinds of battles weren't being fought. I agree with the underreporting and that's why we're leading with it here. And we're going to go through a list of some of the things that might be wrong with, with generative AI.
10:01I think it is an issue. I think Emily, you basically hit on it, right? where you're producing all these tokens when you're going to generate an AI overview that I checked, and you cannot opt out of it. You're correct. Well, you can if you add minus AI to the query. Oh, okay. But you have to do that each time. You can't put a setting somewhere. That's interesting. I didn't know about that. Okay, so you can opt out minus AI, but these things do take more computing than traditional Google search. I guess the argument from these companies would be that they're just going to make their models more efficient.
10:35I mean, we see the increasing amounts of efficiency over time. And, you know, there might be a big upfront energy cost to train, but inference might end up being not that energy intensive. What would you say to that? I would say that we've got Brad Smith at Microsoft giving up on the plans to become a net zero carbon since the beginning of Microsoft. And he said this ridiculous thing about we had a moonshot to get there and turns out with generative AI, the moon is five times further away, which is just an absurd abuse of that metaphor. But yeah, and you see just, you know, Google similarly also backing off of their environment goals.
11:12And so if there really was all these efficiencies to be had, I think they wouldn't be doing that backing off. And I want to also add, I mean, I think this argument about the large amount of training in carbon use on the front end, and then it tapering off with the inference. I mean, this is an argument that straight came from Google. This was, again, in the same paper by David Patterson. I think the title of the paper, I'm not going to get it exactly right, was the cost of training or the cost of generative AI will, probably not generative AI, I think it was the cost of language models will plateau and then decrease or the training costs.
11:53And effectively, the argument being that you have this large investment that we can offset with renewables and then it's going to decrease. But you have to also consider that given that the economics surrounding it, it's not one company training these, right? I mean, it's multiple different companies training these and then multiple different companies providing inference. And so as long as there's some kind of incentive to keep on putting this in products, then they're going to proliferate. So if it was just Google, sure, maybe there might be a case in which there was some kind of planning and there was some kind of way to measure and focus on that and then it actually tapering down.
12:34But you have Google, Anthropic, XAI, of course, OpenAI, Microsoft, Amazon, everyone trying to get a piece doing both training and doing inference. So I think that's, again, you know, like it's hard to put numbers on it, but what we see in this is just a massive investment in this. And that gives a good signal to say that the carbon costs have to be incredibly high. Look, I think it's important for us to, again, to lead here. It's clear that there's some real environmental impacts. and I mean we have Jensen Wang the CEO of Nvidia saying inference is going to take 100 times more compute than than traditional LLM inference and every top executive from these firms that I've asked well is inference going to take more compute it's not exactly as much as Jensen is saying but there is a spectrum so these things are going to be more energy intensive.
13:31And for everybody listening out there, I do think this is important context to take in, that when we talk about AI, there's an environmental cost out there. It's not fully clear what that is, although there is one. And I agree with the authors here that more transparency makes a lot of sense. Now, let's talk about another issue that you bring up in the book, which is benchmark gaming. It's been a hot topic in our big technology discord over the past couple weeks that we see these research labs keep telling us that they have reached a new benchmark or beat a certain level on a new test. And we're all trying to figure out what that means because it does seem like a lot of them are training to the test.
14:15And you have some pointed criticism in the book about the gaming of benchmarks and what that's meant to tell us. So just lay it out for us what's going on with benchmarks and tell us about the gaming, Emily. So, yeah. So when you say the gaming of benchmarks, that makes it sound like the benchmarks are reasonable and they're being misused. But I think actually most of the benchmarks that are out there are not reasonable. They lack what's called construct validity. And construct validity is this two-part test of the thing that we are trying to measure is a real thing, and this measurement correlates with it interestingly.
14:49But nobody actually establishes what these things are meant to measure as a real thing, let alone that second part. And so they are useful sales figures, right, to say, hey, we now have state-of-the-art soda on whatever. But it is not interestingly related to what it's named as measuring, let alone what the systems are actually meant to be for. Yeah, and I would just add that, I mean, there's a lot of work. I mean, prior to the book, Emily and I have spent a lot of time writing on benchmark data sets. And so this has been, you know, like I'm personally obsessed with the ImageNet data set. I'm thinking of another book on the ImageNet data set just from what it entails.
15:31But I mean, you know, the benchmarks, what they purport, there's a lot of different problems in the benchmarks, right? So the construct validity is probably first and foremost. And when we get something where you have something like MedPalm 2 or MedPalm 1 and 2 being measured on the U.S. medical licensing exam, that's not really a test that determines whether one is prepared to be a medical practitioner. practitioner there's so much more involved with being a medical practitioner and uh above and beyond taking the u.s medical license exam you can't take the bar and say you're ready to be a lawyer right i mean there's so much more that has to do with with uh relationships and training and other types of professionalization and there's huge literature in in sociology and sociology of occupations on what professionalization looks like and what it entails and what kind of social skills involved and what that means and how to be adept at being in the discipline.
16:32But then the
16:36kind of different benchmarks are, there's so many different problems just in terms of the way that companies are doing science themselves. They're releasing these benchmarks and often these are benchmarks that they themselves have created and released. So it may be the fact that they are quote unquote teaching to the exam, but they're also, they have no kind of external validity in terms of what they're trying to do. So OpenAI is saying, we had a model that did so well, we had to create a new benchmark for it. Well, who's validating that, right? I mean, even the old benchmarking culture, you had external benchmarks and multiple people would be going to it and saying, oh, we've done better in this benchmark.
17:21Now OpenAI is saying we have our own benchmarks because we did it so well not like the old system was any better but this new system is that well where's the independent validation of this that it says it can do this thing that is purported to say um what do you think about the arc agi test yeah well i mean we spent some time focusing on the arc agi test right the arc agi test it is independent at least that it is it is ostensibly independent i mean it is this is that the the french french show yeah i mean by the way for everybody who's listening it basically asks let me see if i get this right it asks the models to be able to generalize its ability to understand patterns and putting shapes together i think that's the best way to explain yeah so it's it's a it's a bunch of visual puzzles where I think they're all in 2D grids.
18:15And in order to make this something that a large language model can handle, those 2D colorful things are turned into just sequences of letters. And the idea is that you have, I think it's sort of a few-shot learning setup where you have a few exemplars and then an input. And the thing is, can you find an output like that? And when we want to talk about how the names of the benchmarks are already misleading, the fact that that's called Arc AGI, right, that suggests that it's testing for AGI. It's not, it's one specific thing. And I think Cholet's point is that this is something that is a very different kind of task than what people are usually using language models for.
18:54And so the sort of gesture is towards generalization, that if you can do this, even though you weren't trained for it, then that's evidence of something. But if you look at the OpenAI paper-shaped object about this, they used a bunch of them as training data in order to tune the system to be able to do the thing. So, all right, fine. Supervised machine learning kind of works. Right. And the next test, there was ArcGII that came out with a whole bunch of new problems and instantly all the models started doing poorly on those. So let me just ask this. Is there a measure that would allow the two of you to assess whether these AI models are useful or have you just written off their ability to be useful completely?
19:40So useful for what? I mean, you tell me. Well, that's sort of my point is that I think it's perfectly fine to use machine learning to do specific tasks and then you set up a measurement that has to do with the task in context. I'm a computational linguist, so things like automatic transcription are very much in my area. If I were going to evaluate an automatic transcription system, I would say, okay, who am I using it for? What kind of speed varieties? I'm going to collect some data, people speaking, have someone transcribe it for me, a person, and then evaluate how well the various models work on doing that transcription.
20:12And if they work well enough and it is within the tolerances of the use case for me, then great. That's good. Do you believe in the ability to be general? So the ability to be general, and here I'm thinking of the work of Dr. Tamit Gibru, is not an engineering practice. That's an unscoped system. So what Dr. Gebru says is the first step in engineering is your specifications. What is it that you're building? If what you're building is general, you're off on the wrong path. That's not something that you can test for, and it is not well-scoped technology. Yeah. I mean, this notion of generality has always had some specificity in AI, too.
20:52I mean, we mentioned in the book this idea, this is a word I struggle with, and I've taken so many time, but I'm just going to say fruit flies, right? Right, the Josephila, the kind of fruit fly model of genomics, this idea that you have some kind of sequencing that's very common to this one very specific species, right? And there is, in the past, what that's become in AI is the game of chess. It's been game playing, right? I mean, these are very specific tasks. And those don't generalize to something called general intelligence, as if something like that actually exists. I mean, one of the problems in AI research is that the notion of intelligence is very, very poorly defined.
21:41And the notion of generality is very poorly defined or is scoped to what the actual benchmark or what the task is that it is being attempted to achieve. So, I mean, that's, I mean, so this notion of generality is very poorly understood, and it is deployed in a way that is, that makes sense, sound like there is a notion of kind of general intelligence. And it seems to be the fact, I mean, and there's, you know, one of the great paper that we point to in the footnotes of the book is this paper by Nathan Enzmenger, which Enzmenger, that is talking about how chess became the drosophilia of the AI research age and the prior AI hype cycle in the 60s.
22:38East and 70s. And it just happened to be you had a lot of guys that liked chess, and they wanted to compete with the Soviets who had chess dominance, right? And so those tasks become kind of these tasks about like, well, these are the things we kind of like. And we're actually seeing some of that, again, it's like, well, these are tasks that we think are suitable. These are tasks that are scoped in a way that we think are the most worthwhile problems, but they're not tasked to think about, well, what exists in the world that is going to be helpful and useful in scope to specific execution, right?
23:12This notion of an everything system is wildly unscoped. But, okay, so it is unscoped. But I think everybody listening or watching right now would probably say, well, just my basic use of ChatGPT, it can tell me about history, it can write a poem, it can create a game. Okay, I see Emily reacting already. It can search the web and give me plans. can do all these different things in these different disciplines. So there is, I think for people listening, there would be a sense that there is an ability to go into various different disciplines and perform. And whether you say it's a magic trick or not, it's clear that it can.
23:53And so what I guess I'm trying to get at is, I mean, is there a way to measure that or do you think that that is in itself a wrong assertion? So, yes, I think it's a wrong assertion. What ChatGPT can do is it can mimic human language use across many different domains. And so it can produce the form of a poem. It can produce the form of a travel itinerary. It can produce the form of a Wikipedia page on the history of some event. It is an extremely bad idea to use it if you actually have an information need. setting aside the environmental impacts of using chat GPT and setting aside the terrible labor practices behind it and the awful exploitation of data workers who have to look at the terrible outputs so that the consumer sees fewer of them.
24:47And by terrible outputs, I mean, violence and racism and all kinds of sort of psychologically harmful stuff. We've covered that on the show. Yes. What's that? No, we've had one of the people who've been rating this content on the show. Folks who are interested, I'll link it in the show notes. Richard Matanga was here to talk about what that experience was like. Sorry, go ahead. So setting aside all of that, if you have an information need, so something you genuinely don't know, then taking the output of the synthetic text extruding machine doesn't set you up to actually learn more on a few levels.
25:22If you don't already know, you can't necessarily quickly check, except maybe doing an additional search without ChatGPT, at which point, why not just do that search? but also it is poor information practices to assume that the world is set up so that if I have a question, there is a machine that can give me the answer. When I'm doing information access, instead what I'm doing is understanding the sources that that kind of information comes from, how they're situated with respect to each other, how they land in the world. And this is some work I've done with Shrag Shah on information behavior and why chatbots, even if they were extremely accurate, would actually be a bad way to do these practices.
26:01So just to, you know, back to your point, yes, this system is set up to output plausible looking text on a wide variety of topics. And that's, therein lies the danger, because it seems like we are almost there to the robo-doctor, the robo-lawyer, the robo-tutor. And in fact, not only is that not true, not only is it environmentally rudeness, et cetera, but that is not a good world to live in. And thinking about, thinking about, I just want to hit on this point. This is, I agree, I disagree with you on this one. I do think that some of the points that you're making are well founded. We don't want these things to be lawyers right away.
26:36But let me at least point you to one use that I've had recently, and you could tell me where I'm, where I'm going wrong, if you think I am. I mean, I'm in Paris now, a little work, a little vacation at the same time. And what I've done is I've taken two documents that I've had friends who they have they've been here often they put together documents that they send to friends when they go here I've uploaded that into chat GPT and then I have chat GPT like search the web and give me ideas of what to do I tell it where I am I tell it where I'm going and it searches through like for instance like all the museums the art galleries the festivals the concerts and it brings it into one place and that's been extremely useful to me to find new cultural events, concerts.
27:18There's even a bread festival going on here that I had no idea about. And now I'm going to go because it's found it for me. So there's a link when it comes to this stuff, there's a link that you can go out and double check the work. But as far as finding information on the web, the fact that it's able to go and comb the internet for these events and then take into context some of the context that I've given it with these documents, I think is very impressive. And that's just one use case. So I'm not asking it to be a lawyer. I'm kind of asking it to be what you said, an itinerary planner. What's wrong with that?
Read the full transcript
27:53So, I mean, first of all, you have these lovely documents from your friends. And I guess what you're saying is missing is whatever current events are. So they've given you some sort of like, these are general things to look for, but they haven't looked into what's going on right now. What's wrong with that? You know, on several levels, what would we do in a prior age, like even pre-internet, right? The local newspapers would list current events. Here's what's going on. If you landed in a city, you would go find the local, probably local indie newspaper and look up the events page. And that system was based on a series of relationships within the community between the people putting on festivals and the newspaper writers.
28:31And it helps support probably the local news information ecosystem, which was a good thing. But on top of that, if something wasn't listed, you could think about why is this not listed? What's the relationship that's missing? Your chat GPT output is going to give you some nonsense. And you're right, this is a use case where you can verify whether this is real or not. It is also likely going to miss some things. And the things that are not surfaced for you are not surfaced because of the complex set of biases that got rolled into the system, plus whatever the roll of the die was this time. And anytime someone says, well, I need chat GPT for this, usually one of two things is going on.
29:16Usually it's either there's another way of doing that, that is giving you more opportunities to be in community with people to make connections, or there is some serious unmet need, which doesn't sound like it's this case. And if we sort of pull the frame back a little bit, we can say, why is it that someone felt like the only option was a synthetic text extruding machine? And here, I think you've fallen into the former of these, which is what are you missing out on by doing it this way? What are the connections you could be making to the people around you? If you're staying in an Airbnb, maybe the Airbnb host, if you're in a hotel, the concierge, to get answers to these questions when you're looking to the machine instead.
29:54I would also say this is, I just want to add, I mean, I would also say this is a pretty like low stakes scenario, right? You can go out, you can verify these things. You can go to existing resources of event calendars that people also spend a lot of time curating online. I mean, there's a lot of stuff that's already curated online. And I mean, it's not like this exists in prior incidents of technology. I mean, one of the people that we cite in the book and talk a lot about is Dr. Sophia Noble's work on Google and the kind of way that Google results present very violent content with regards to racial minorities.
30:34One of the parts of the book that I like to reference and that a lot of people don't reference initially is this kind of part that she talks about. She talks about Yelp, and she talks about Yelp and specifically and what it's referring in terms of a black hairdresser and the way that Yelp effectively was shutting this person out of business because there was specific need that she had for black residents of the city that she was studying and braiding hair and doing other black hairstyles, right? And so this is kind of a function of all kind of information retrieval systems, right? You think about what they're including, what they're excluding, right?
31:17So this is not very consequential here, but in any kind of area of, say, summarization or any kind of retrieval, you do need to have some kind of expertise where you can verify that and ensure that what's getting in there is not missing something huge and you're it's going to basically then take this set of information access resources or systems in this case crawling the web and and knowing that that's going to miss something and then it's going to exacerbate that because then you cannot situate those sources in context okay let me just give my counter argument and then we can move on from this uh my counter argument would be a couple things First of all, I don't speak French, so the local newspaper would kind of be lost on me.
31:58I am speaking, okay, so I am staying at a residence place. We swapped apartments. So she's in my New York apartment. I'm here. So maybe she and I could have gone over that newspaper together. That's fair. But the newspaper, speaking of things that leave stuff out, the newspaper leaves stuff out all the time. It exercises editorial judgment. So it is bot editorial judgment for newspaper editorial judgment. but the bot can be in some ways more comprehensive because it's searching the entire web uh and i'll just say one last thing about this i never felt i didn't feel the need to use it um i didn't say i need to use it to figure out what's going on like again i had these documents what's useful about it is speaking of making connections with the local community if i'm able to here's the word be efficient uh in my research through using it i could spend much more time uh out in the community versus searching the web or reading the newspaper so what's your thought on that on those arguments um sorry so that i'm getting distracted by alex's cat walking around listeners alex's cat is here alex what's your cat's name this is this is clara and i'd lift her up but my uh i have a shoulder injury.
33:11But she is, she's knocking the mic around. So I'm going to not, I'm just trying to keep her off the mic. Yeah. Yeah. Thank you. So, you know, the efficiency argument. So this is efficiency argument in the context of leisure activities, as opposed to in the context of work. You mentioned along the way that it is searching the whole web for you. You don't know that, actually. That's right. Yeah. And also the whole web includes a lot of stuff that you don't actually want. Like lots and lots and lots of the web is just garbage SEO stuff. And maybe you're seeing more of that in your chat GPT output than you would with a search engine, which as Alex mentioned, also has issues.
33:52And then finally, I'm going to take issue with you. SEO garbage is made for the search engine. It is, but the search engines also in order to stay in business have to be fighting back against the SEO garbage. It's a constant battle. Probably the chatbots as well. So you mentioned newspaper editorial judgment versus bot editorial judgment. And I'm going to take issue there because a bot is not the kind of thing that can have judgment, nor is it the kind of thing that can have accountability for exercising judgment. And so I think that, yes, this Alex is saying this is low stakes, but if you're using it as sort of a motivation for these things being useful in the world, then you have to deal with the fact that the useful in the world is going to entail many more higher stakes things.
34:32And then we really have to worry about accountability. I would also want to say too, I mean, there's a lot of, I think this argument from like quote unquote capabilities, which I don't know really what that term means. And that's another poorly defined term, I think, especially when it comes to AGI. But I mean, this argument from kind of like, well, I find it useful. I don't find terribly convincing, right? I mean, it's sort of like, well, okay, you have found it useful in either a situation in which there is a way to have some kind of verification of sources that you know about and have some kind of ground truth about, or you found it useful kind of from a variety of this, of these different situations.
35:23But if I'm asking a chatbot about something about an area that I know quite a lot about, say, sociology or social movements literature, I then have that knowledge to verify that just from my social skill in that area. And this is a term I'm kind of borrowing from a sociologist, Neil Flinkstein, and my knowledge of how to navigate those areas and my professionalization as a sociologist. Okay, but then once it gets into those areas in which verifiability just escapes me, which is most areas, because we're not professionals in those areas. And although a lot of us want to be jacks of all trade, jacks and jills of all trades, then we lose that ability and we don't have the social skill or depth of knowledge to verify that in the same way.
36:15And so I'm really not convinced by those. Well, these are useful for me in these pretty low state contexts because that slippage then means that we're going to miss some pretty big things in some really dire contexts. okay well let's turn it up a notch when we come back because we're going to talk about ai at work and ai in the medical context and maybe we can even touch a little bit on doomerism which you write about in the book and uh and there's plenty else on the agenda so we'll be back right after this and we're back here on big technology podcast with professor emily m bender and alex hannah they are the authors of the ai con how to fight big tech's hype and create the future we want here it is so let's go to usefulness and we'll start with a generative ai in the medical context because why don't we just go straight for uh the example that we'll probably have the biggest disagreement on here and i'm not saying that i think generative ai should play the role of a doctor in fact when i wrote my list of things i agree with you both on i don't think that ai should be a therapist at least not yet and we know now that AI is the number one use according to a recent study is companionship and therapy and the therapy side really scares me and I think the companionship isn't the best thing in the world either but in medicine I do find that there is some use for it medicine is a field overrun by paperwork and insurance requirements that I think keep have have ruined the healthcare system because they keep doctors effectively tied to their computers, writing notes, as opposed to seeing patients or living their lives.
38:04And Alex, before the break, you mentioned that one of the areas that this stuff is useful is when it starts to operate in your area of expertise, because you're able to verify that. So, I mean, we're going to go with one use that I find to be pretty good here and would sort of to me doesn't make a generative AI feel like a con is when a doctor is seeing a patient and they can put a transcription take a transcription of the conversation that they have with the patient and then have AI synthesize what they talked about and summarize it and put it into the systems that they have for electric medical records and then verify that so they don't have to spend the time writing those summaries up and can actually go and spend some more time with patients.
38:53So what's the problem with that? There are so many problems with that. And the first thing I want to say is that you named the underlying problem when you talked about insurance requiring so much paperwork. So this is one of those situations where there's a real problem here. It's not that doctors shouldn't be writing clinical notes. That is actually part of the care. But there is a lot of additional paperwork that is required because of the way insurance systems and especially the one in the United States are set up. And so we could work on solving that problem. And this is a case where sort of the turn towards large language models, so-called generative AI as an approach to this is showing us the existence of an issue.
39:32But that doesn't mean it is a good solution. So many problems. One is writing the clinical note is actually part of the process of care. It is the doctor reflecting on what came out of that conversation with the patient and thinking it through, writing it down, plans for next treatment. That is not something that I want doctors to get out of the habit of doing as part of the care. Now, they might feel like they don't have time for it. That's also a systemic issue. Secondly, these things are set up as like ambient listeners, which is a huge privacy issue. As soon as you've collected that data, it becomes sort of this like radioactive pile of danger.
40:08Thirdly, you've got the fact that automatic transcription systems, which are the first step in this, do not work equally well for different language varieties. So think about somebody who's speaking a second language. Think about somebody who's got dysarthria. So an older person whose speech isn't very clear. Think about a doctor who is an immigrant to the community that they're working in, who's got extra work to do now because their words are not well transcribed. And so the clinical notes thing doesn't work well for them. But the system is set up where there's these expectations that they can see more patients because the AI, in quotes, is taking care of all of this for them.
40:43And there's a beautiful essay that came out recently, I think, in Stat News. And I was looking for the name of the author. Didn't find it real quick. Really reflecting on how important it is to her that the doctor do that part of the care, of actually pulling out from the conversation, this is what matters. And it's not just simple summarization. It is actually part of the medical work to go from the back and forth had with the patient, all of the doctor's expertise to what goes into that note. Yeah. So I want to add on, as Emily has said, so much of what I want to get at, which I think is, but I have, I think, three or four separate points in addition to that.
41:20So first off is the technical point. So there's tools that are purported to be summarization. There was some great reporting by Garen Spirk and Hilda Shellman and the AP from last October that was looking at Whisper specifically. So that's OpenAI's ASR system, automated speech recognition system, that said that medical transcription basically was making up a lot of shit. And then we knew that they had, quote unquote, hallucinations. Again, that's not a term that we use in the book. We say that it's, I say it's making shit up, but that is maybe even granting too much anthropomorphizing of the system for me.
42:04And so, but there is a lot of these things. Quoting from that text, some of that invented text includes racial commentary, violent rhetoric, and even imagined medical treatments. So that's one major problem. The second problem is that medical transcription has been this area, which has been an area in which medicine has been forcing kind of this casualization of work for years. Right. And so medical note taking that exists in hospitals now, much of that is done remotely. So it's gone and taken this work that has been seen as kind of like this busy work or this thing that like, I don't want to write up my medical notes to be this type of work that needs to be foisted on someone else.
42:50So prior to this kind of ASR element of it is we've had these, oh, thanks for linking that, Emily, and I'll link the AP article that I'm looking at too. Part of that work has actually been offshored a lot into this kind of movement of outsourcing. So a lot of that is done remotely as a part of this casualization. And the scene to be, I think, to be a lot of, I want to point out the gender notion of this. This is like a very kind of like women's based work. And that reflects a lot of the ways in which so much of quote unquote AI technology wants to basically take the work that has been traditionally the domain of women and is saying, well, we can automate that or we can casualize that in different ways.
43:41And that's important because it sees this work as not actually part of, quote unquote, the work. It is seen as work that ought to be casualized and offshored. and so and I appreciate the essay that Emily shared because that essay is saying like no this is actually part of the element of doctoring and then I want to also just kind of couch all of this stuff in the kind of political economy of the medical industry thinking about what does it mean to rush and put and have more and more remote medicine having more and more doctors see more patients and these efficiency gains from doctors isn't going to like make their jobs necessarily easier, it's going to put more of a pressure on them.
44:25Now that you're in a position where you don't have to take medical notes, you're going to be running from position to from appointment to appointment to appointment. And my sister is a nurse, she's a nurse practitioner. And she's basically seeing this in her job right now at her clinic. She's like, now we have these things where I have to see patients now, you know, and if you know, it's not gonna I'm gonna go and be on the beach anywhere, it means that I'm going to have, you know, I'm going to have nine to 10, 15 minute appointments a day, I'm not going to have enough proper treatment, proper time to spend with patients.
44:59So these things could be, you know, like I would say to the coda to all this is that if AI boosters could really offshore all of doctoring to chatbots, they would and this is one case in which Bill Gates has said, you know, in 10 years, we're not going to have teachers and doctors, what a nightmare scenario to have non-teachers and not doctors and greg carato really gives it away and we cite him a book where he says a med palm too you know this thing is really efficient we're going to increase tenfold our medical ability but i wouldn't want this to be part of my family's medical uh medical journey okay but you're again here you're you're picking out what what is like some of the most extreme statements and i i started my question saying it's bill gates and Bill Gates can make extreme extreme statements I don't think that guy I don't think he's the guy and I think that that that doesn't reflect the broad consensus here and definitely not the question that I asked which again was about using using this to take some of the time that the doctors are using you know in paperwork and give that back to either the doctors or to have them be able to see more patients so very much address that point first I want to name the author of that essay.
46:13Her name is Alia Barakat, and it's a beautiful essay. She's a mathematician and also a patient with a chronic condition. Wonderful essay. But yeah, you said, give that time back to the doctors or have them see more patients, right? It is not going to be going back to the doctors. That's not how our healthcare system works. And it's also going to therefore decrease quality of patient care. It is lose-lose, except for the hospitals maybe getting more money and certainly the tech companies that are selling this to the hospitals. Okay. I'm also curious in terms of thinking about it. I mean, yes. I'm curious in thinking about the more nuanced position and who are the reference here that you're thinking of, Alex?
46:52What's the consensus on this? Because we see the egregious elements of this, and I'm wondering what the medical consensus is. What's an example, just to put poison, now I'm interviewing you, but like who's someone that do you think is doing this very well on well i mean someone doing this well like again i don't think that this stuff is well developed yet uh but i've definitely seen enough doctors just buried in paperwork uh and we we started this whole segment talking about how this is i guess it's an insurance driven thing um and so uh it's interesting that i guess do you both not like the way that the insurance companies are guiding the system but also think that it's good practice to have doctors write those notes for them or hold on there's two use cases for doctor's notes right there is actually documenting for the patient and for the rest of the care team what has happened in this session and that i think is a super important part of the work of doctoring i believe you that there's a lot of additional paperwork that has to do with getting the insurance companies to pay back and no i don't like that system at all that it does not the insurance companies are not providing any value they are just vampires on our health care system in the U.S.
48:04Okay I think we can we can agree on that front and anyway but I do I do think that as this stuff gets better I understand a patient wants this to happen do I think a doctor would be giving them worse care if they allowed the AI to summarize the notes or to pick out the more important parts if this stuff was working well not necessarily. So that's a that's a big if you know what does it mean when this stuff is getting better and this stuff working well do you mean kind of like the absence of making shit up right definitely i mean but we all we both we all agree that the doctor will have to verify and check this information after well i guess the problem was there then like then why are we having the doctor double check that to begin with right in an area where the doctor has 15 minutes to see every patient and there is an ai quote unquote scribe doing or quote-unquote ai i don't want to call it ai scribe there's an automatic speech recognition tool right doing automatic speech recognition on these things in what in what space or with what time does the doctor have to verify those in an area i mean this is i mean like well the time that they would be spending writing those notes in the first place is verification an easier test and transcription?
49:24I guess that's my question. I would proffer no. I mean, just from my experience using these systems. And I mean, I'm not a doctor. Thank God. Although, although I've thought about it. Not that kind of doctor, to the chagrin of my parents. But then, I guess the question is, of that is, in my experience, I've used these tools for interviews, specifically in kind of qualitative interviews with data workers, and have spent time with these tools, and have just had such an awful time trying to think about this, especially with regards to, you know, this isn't, you know, this isn't medical terminology, but it's terminology around doing data work.
50:08We're talking about training AI systems, and it just, it is such a terrible job, and at one point, I threw it all out, and I said, okay, I just am sending this to somebody to actually transcribe because this is not helpful for me. And it's taking me more time, starting with the transcript and then from doing it from scratch. And I've transcribed, you know, I'm not a primarily qualitative interviewer, but I've spent time, you know, transcribing dozens of interviews in my research career and have found it just very difficult. So, I mean, I guess the question is, is that verification, you know, is that taking the time that could be just be used for the doctoring and working with patients.
50:50And I mean, like, you know, holding, you know, everything about the insurance industry, you know, stable, you know, like, is that kind of notion of thinking about, you know, different patients, how the patient presents, how the patient is describing their, their, their, how they're, you know, how they're presenting is that, you know, that is often the work of doing it, you know, and the medical training I do have is that I am at one, I was at one point a licensed EMT. And writing up PCRs is not like, you know, no one wants to write up the PCRs. At the same time, you're spending time taking note of how a patient is presenting the patient is, you know, Alex rhythmic, just bringing it back to the Alex's, the, you know, the patient is cyanotic around their lips.
51:40Like these are things that, you know, a healthcare professional would be paying attention to is making notes maybe because they're like writing it later. So I'm thinking about this process of writing and what it does to our own practice of viewing and aiding and administering medical care. Okay. I mean, we'll agree to disagree on this front. But again, I think we are all on the same page that insurance companies requiring additional writing just because they hope you don't ever get to the claim if you don't file it that's probably bad uh and we don't think that there should be uh that that there should be ai doctors at least yet i that's what i say i think you guys probably say never so all right i want to end on this which is or maybe maybe we can do two more topics um I guess like here's my question for you.
52:30A lot of the discussion of AI's usefulness in jobs in the book discusses these tools being imposed top down. But what if they come bottom up? Like what if a worker can find use for them and actually make their job easier by getting good at using something like a chat GPT or a Claude? Or, you know, again, we kind of talk through the medical use case. If a doctor does find that this is useful for them, are you opposed to that? So, yes. And I think that actually Cadbury, of all people, put it best. There's this hilarious commercial that was for the Indian market, sort of showing how the supposed efficiencies that you're getting out of this just ramps up the speed of things and doesn't leave you time to really get into the work that you're doing and be there.
53:18I think that the most credible use cases I've heard for these things are, first of all, as coding assistants. So that's sort of a machine translation problem between natural language and some programming language. And there I really worry about technical debt, where you have output code that was not written by a person that's not well documented, that becomes someone else's problems debugged down the line. But also in writing emails. People hate writing emails and people hate reading emails. So you get these scenarios where somebody writes bullet points, uses ChatGPT to turn it into an email, and the person on the other end might use ChatGPT to summarize it.
53:57And it's like, okay, so what are we doing here? And again, taking a step back and saying, what are the systems that are requiring all this writing that everyone finds a nuisance to write and to read? Can we rethink those systems? And also, I just have to say that whenever I'm on the receiving end of synthetic text, I am hugely offended. it and one of the things that we put in the book is i definitely got one of those emails yesterday and i was like you use chat gpt for this i know you did yeah if you couldn't be bothered to write it why should i bother to read it right yeah and i mean i mean it is it's very interesting putting this and like thinking about cases in which workers are using this kind of organically and i kind of like in the case where it's you know like this is the case where like first off i've I've heard a very little of that personally, especially for professional.
54:45I mean, I think there's plenty of workers that are finding a lot of use of this, but. I would say the analog that I find to be where it's not top down is in education. And to that degree, I think that's kind of a failure in thinking about like what education is, right? I mean, in that case. Well, for students to be using this to get through their classes. Yeah, right. Are you talking about teachers putting stuff together? Well, both. But I'm thinking about the students, right? And I'm just thinking about areas in which, but I'm using that as sort of an analog and then thinking about what are the conditions that are forcing students to use this, right?
55:18If there's kind of cases in which this seems to be sort of useful, okay, what are the cases in which, what does that say about the job? What does it say about like how the work is oriented, right? In that case, then maybe there might be, needs to be kind of different efficiencies or thinking about how the job is operating, right? I then worry then that these things become mandated in work environments. They say, well, people are using this and so everybody's using this. And then where does that leave the people who are resistors or thinking about, well, I know this can't do a good job. So where's that putting me?
55:53And I think we've already seen such a justification for this as being a place where employers have been reducing positions by the scores because there's a notion that these tools can do these jobs suitably and to a certain kind of degree of proficiency, which is just not the case. that has me worried about down the line in these areas that Emily's mentioned, the kind of technical debt area, the kind of how do we know? And there's kind of an overestimation of capabilities of these tools in that case. Okay. I know we're at time or close to time. Can I ask you one question about doomers before we get out of here?
56:34Sure. Let's end by dogging on the doomers. Okay. So I definitely, I saw that there was a chapter about doomers here and I was excited to read it because my position has been largely that those who are worried that large language models are going to turn us into paper clips or either marketing what they're selling or just very into, I don't know, they like the smell of their own body odor. I mean, I guess it's not a terrible thing to be worried about, but there's so much more. And it seems so unlikely that this is going to hurt us. So I definitely wanted to get your take on why you're down on doomerism.
57:14And let me just give my one caveat here. There's a line in your book that says that AI safety is just doomerism and it's only about these long-term problems. But I've definitely heard some of the AI safety folks like Dan Hendricks from the Center of AI Safety talking about really important near-term issues like whether this technology could help virologists with bad intent. So I wouldn't malign the entire AI safety field. But the doomerous stuff, I hear your point. All right, so attack that and then we'll get out of here. So I just want to put in a shout out for a new book by Adam Becker called More Everything Forever, which really goes deep into the connections between the sort of doomerous thought and these more palatable looking sides of what's called effective altruism.
58:08And also in that context, there's a wonderful paper by Tamit Gibru and Emil Torres on what they call the Tescrial bundle of ideologies. And, you know, I think that if you're if you're concerned about the systems is not rooted in real people and real communities and things that are actually happening, like even this like, oh, but bad actors could use it to, you know, more quickly design, you know, viruses and stuff like that. That's still speculative. Right. So anytime we are taking the focus away, it's like, has that happened? Right. This is this is still people writing science fiction, fan fiction for themselves.
58:47And, you know, it's not it's based on these jumped up ideas about what the technology can do and taking the focus away from the actual harms that are happening now, including the environmental stuff we started with. Right. But yeah, I mean, I will. I will say a virus. Right. You want to get you want to get ahead of that. Right. Like we had it with social media. There were some issues with social media, but some of these, there was not a focus on some of like the potential long-term issues. And that only came up later on, at least in the beginning. You don't agree. There are problems with social media, for sure.
59:22And some of those problems were documented and explained early on and people were not paying attention. but they were real problems that were being documented as they were happening as opposed to imaginaries about well someone's going to use this and dr evil up a bad virus okay yeah go ahead alex for the sake of time i think that's fine i don't have much to add all right well look the book is called the ai con how to fight big tech's hype and create the future we want the authors are Emily M. Bender and Alex Hannah Emily and Alex I've been reading your work for a long time and it's great to have a chance to speak with you like I said at the top you know for those who are listening or watching and you may not agree with everything either everything I said everything our guests said hey at least you know you know these arguments and you know the arguments for and against and we trust you to to make up your own opinion and do further the research and we've definitely had plenty of good stuff to keep digging into shout it out over the course of this conversation so emily and alex great to see you thank you so much for joining the show thank you for this conversation and enjoy paris thanks alex have a great time thank you both thank you everybody for listening we'll see you on friday for our news recap until then we'll see you next time on big technology podcast
1:00:49Thank you.
From the publisher
Emily Bender is a computational linguistics professor at the University of Washington. Alex Hanna is the Director of Research at the Distributed AI Research Institute. Bender and Hanna join Big Technology to discuss what their new book, “The AI‑Con," which they describe as the layered ways today’s language‑model boom obscures environmental costs, labor harms, and shaky science. Tune in to hear a lively back‑and‑forth on whether chatbots are useful tools or polished parlor tricks. We also cover benchmark gaming, data‑center water use, doomerism, and more. Hit play for a candid debate that will leave you smarter about where generative AI really stands — and what comes next.
---
Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice.
Want a discount for Big Technology on Substack? Here’s 25% off for the first year, which includes membership to our subscriber Discord: https://www.bigtechnology.com/subscribe?coupon=0843016b
Questions? Feedback? Write to: bigtechnologypodcast@gmail.com
Learn more about your ad choices. Visit megaphone.fm/adchoices
