How today’s AI giants mirror global empires of the past, with journalist Karen Hao

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:52When the term artificial intelligence was coined in the first place, it was coined largely as a marketing ploy because this Dartmouth assistant professor, John McCarthy, he said many years later, I invented the term artificial intelligence. I invented it because we had to do something when we were trying to get money for a summer study. And his mentor actually heavily pushed back against that term, saying no one's going to know what that means. It's going to oversell this technology. It's going to create all this confusion. And of course it did. But also the term ended up just sticking. And now we have the term AGI replicating all the same problems and confusions.

1:40Karen Howe wants to cut through some of that confusion. Her new book, Empire of AI, chronicles the rise of generative AI through the lens of the Goliath companies behind it. She likens their rise to imperial empires of the past. After years of reporting on OpenAI, Karen lays out the history of the company and offers her own critique about where AI is headed. Her book is a must-read for AI skeptics and enthusiasts alike, and I'm so excited to share our conversation. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

2:28Hi, Karen. Welcome to Pioneers of AI. Thank you so much for having me, Rana. So we share the MIT connection. And back in the day when you were at MIT Tech Review, you covered my company Affectiva a few times. So it's great to reconnect after all these years. Yeah, it's really great to speak again. Okay, we are going to talk about your book, Empire of AI. Congratulations. And of course, the book became an instant New York Times bestseller. So yay, congratulations. How are you feeling? Really, yeah, really grateful for the reception that it's been getting. It's what I dreamed of. So it's amazing to see it happening.

3:05So I have to ask you this. Is this all going to become like a Netflix TV series or something? I can't wait for it. That would be my dream. Any Netflix producer listening to this, call me. Exactly. All right. Let's see if we can make it happen. And, you know, one of the things I always do when I pick a book is I go to the acknowledgement section and I read that first. And I was actually really stuck by yours because you start with this idea of belief, this leap of faith that we all take when we are, you know, when we have deep conviction in an idea or a product or or even ourselves. Right. So I want to ask you this.

3:43Why is the concept of belief so central to both your personal story and the theme of the book? Yeah, I was really struck by the fact that throughout my career, I have had just amazing people around me that have really believed in me. And that has been such a core aspect of being able to do the things that I have wanted to do. But it is also such a central part of opening eye story in that it can also become a toxic thing. I opened the book with this quote that Sam Altman found when he was a young entrepreneur and he ended up writing about it in a blog post in 2013. And it goes like this. Successful people build companies.

4:30More successful people build countries. The most successful people build religions. And then he's reflecting on that, saying that the most successful founders don't seek to build companies. They seek to build religions and ultimately building a company is the best way to do so. And so it felt like belief is such a core theme that runs and courses through OpenAI's history as well. Just belief in AGI, belief in oneself, belief in a religion. It's such a fascinating core aspect of the OpenAI story, but it is a double-edged sword sometimes. Yeah, absolutely. And actually, one of your primary arguments in the book is that these big AI companies like OpenAI are modern day empires and you compare them to the European imperial powers that colonize so many countries.

5:25Walk us through this argument and why do you make this analogy? Yeah, so there are four parallels that I draw between empires of AI and empires of old. The first one is that empires lay claim to resources that are not their own, but they reinterpret the rules to suggest that those resources were always their own. And that refers to the way that companies scrape the data on the Internet and they say that it's totally fair game because it's in the public domain. But of course, the people who put their data on the Internet, they didn't give informed consent for their data to be scraped and used to train models that ultimately might constrain their future economic opportunity, for example.

6:03It also refers to the fact that these companies take the intellectual property of artists, writers, and creators without credit or compensation and say that it's fair use under copyright law. Empires also exploit a lot of labor, and that refers not just to the labor exploitation that happens in the AI development process, which I document extensively in my book, but also the fact that the production of this technology, this technology is inherently a labor-automating one. In that Opening Eyes definition of AGI is highly autonomous systems that outperform humans in most economically valuable work.

6:41So they explicitly say that they are trying to build systems that are automating away the tasks that people usually get paid for. So that in and of itself then erodes workers' ability to bargain, to negotiate, and that's going to lead to labor exploitation as well. The third feature of Empire is that they monopolize knowledge production. So in the last 10 years, what we've seen is the AI industry has become so resource-rich that they're able to offer compensation packages that easily cross over a million dollars. And so the top AI researchers in the world have moved from working for academia or independent institutions to working for these companies.

7:24And the effect would be exactly the effect that you would imagine if most climate scientists in the world were bankrolled by oil and gas companies. You would not get a clear picture of the climate crisis. And that is essentially what's happened with AI research. Companies are actively censoring the research that is inconvenient to them. So we are not getting a full picture of the true limitations of this technology as it's being deployed. into the world. And then the fourth and final feature of empires is that they always have this narrative that there are good empires and evil empires in the world.

8:04And they, the good empire, have to be an empire in the first place to be strong enough to beat back the evil empire. And throughout my book, I document how OpenAI has always had an evil empire, but they change who the evil empire is depending on what's convenient. So in the beginning, Google was the evil empire. Now, increasingly, China is the evil empire. And this idea is that if the evil empire gets hold of this technology first, then humanity will go to hell. But if they, the good empire, have unfettered access to resources, to labor, and they can get this technology first, then they will be able to civilize the world, bring progress and modernity to all of humanity, and humanity will ultimately have a chance to go to heaven.

8:52Yeah. And that comes back to this like strong aspect of narrative and storytelling and belief, right? You have to have this strong belief around this very kind of important mission. You know, one of the things that really struck me reading this book is the cast of characters in this empire, right? And it shouldn't be surprising to me because I've been in this industry for so long, but it still was really jarring that most of the book, there was just a lack of diversity and specifically a lack of diversity around women, right? I just find that really, yeah, it's dismaying. So I would love to hear, I don't know, your thoughts.

9:31And it's not just at the CEO level. It's really kind of the people building and shaping these key technologies and making the key decisions too. Yeah. I mean, this is absolutely like OpenAI sits at the intersection of the AI research world and Silicon Valley. Both worlds are ones where women just do not rise to the top. In AI research, I think the last stat I saw was only 12 % of AI researchers are women. And in Silicon Valley, I think the most recent stat was only 2 % of founders that receive investments from VCs are women. And so automatically you have these two deeply male-dominated sectors or fields merged into open AI, which then ultimately reflects that challenge where women are not rising to the top.

10:27And part of it is there's a lot of hostility in these environments towards women. And another part of it is that women self-select to leave because they don't agree with the general premise of what OpenAI or these other companies are pursuing, this idea of artificial general intelligence. So I personally have a number of friends who are AI researchers that chose not to go into the industry in part because they just think that this quest to try and recreate human intelligence with this idea of a zero-sum game, winner takes all, is just not the right one. And they observe that this kind of quest typically ends up being the most harmful for marginalized communities, including women.

11:13And so I think there's a lot of intersecting reasons why ultimately it is men that predominantly shape this technology. I was also struck by the importance of this idea of a network effect, right? So for example, Sam Altman, he is an investor in many of the companies across the AI tech stack, everything from fusion to health companies. And they're always like, you know, it's a group of people, they're always supporting each other, investing in each other's companies. Is that kind of an important aspect of empire building as well? Absolutely. So this idea of network effects is something that Altman learned from his mentors that in order to aim for monopoly and create a dominant position in the market for anything that you're building, you want to interlock both the people that you know and the investments that you know so that you can continue to gain more and more leverage in that marketplace.

12:17And so this is exactly the playbook that Altman then ends up using. He says many times throughout his career that one of the best pieces of advice that he got early on was this idea of building network effects. He is intentionally trying to interweave both his network of people and his network of companies to be as tightly integrated as possible so that it turns into a fortress, into an empire that is impenetrable. In a minute, we break down the AI empire. We dig into what data extraction and labor exploitation actually looks like on the ground, and we get into how we can build AI in a different, more ethical way.

13:01Stay with us.

13:17If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's Teamwork Collection. It has planning in JIRA, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger.

14:06So I want to kind of dig into this idea of extractive AI and this key element of empire building, which is extracting resources, whether it's natural resources or labor or data, right? And I love this line. Data is the last frontier of colonization. And I think at this point, we all know that LMs are very data hungry and kind of often scraped off the Internet. But let's dig into that a little bit more. Like, what was some of the key lessons you learned and how these companies are approaching access, not just to the quantity, but also to the quality of the data? Yeah. And that quote was from Kioni Mahalona, who is an indigenous researcher and journalist and jack-of-all-trades person that works for this nonprofit Tahiku Media in New Zealand or Aotearoa.

14:58And he was saying this to me as like, this is, you know, as an indigenous person, this is, it's so blatantly obvious that before people used to take our, colonizers used to take our land and then sell it back to us. And now they're just taking our data and turning it into a service and selling it back to us. And it's like there was never any consent along the way. So originally, before OpenAI really started to dominate the scene, the AI research field and the industry was actually shifting more towards tiny AI and this idea of curating data sets and really making sure that it's clean, pristine data that you're using to train a model.

15:37And ultimately, that means you can get away with really, really small data sets that produce quite powerful models and also very predictable models in terms of their behavior, because you know exactly what you're feeding into the system. But what OpenAI did was they started going for this large scale scraping of the Internet. And when you put everything from the Internet into your training data, then you also get all of the bad stuff. So there's plenty of gunk that gets left in. And that is part of what leads to downstream labor exploitation. And I interviewed this one executive of the company Append, which is a third party platform that connects companies like Opening Eye with contractors in the global south or in economically vulnerable communities that do data preparation and data cleaning.

16:29And he said before, we used to clean the inputs and now we put everything in and we control the outputs. That's been the paradigm shift of the last few years. And what controlling the outputs ultimately means is that there are workers who have to perform the grotesque work of content moderation. Because when you have a text generation model that can spew anything and is trained on the worst parts of the Internet, it is going to start spewing really toxic, hateful content. And so to block that from ever-reaching users, you need to wrap these models with filters that prevent this content from being exposed to the user.

17:12And that has downstream ramifications for the people that do that work. One of the stories that really struck me in the book is the story of Oscarina. Can you share that with us? Because I think it really drives the labor that goes on behind the scenes in terms of building AI. It really drives it home. Yeah, so Oscarina was someone that I met actually pre-generative AI era when the AI industry, of course, already was relying very heavily on data annotation and data preparation. And she was a Venezuelan refugee that lived in Colombia. And the reason I went to go meet her is because Venezuela specifically became this huge hotbed of data annotators in the 2016 to 2020 era because the country was undergoing the worst peacetime economic crisis in 50 years at exactly the same moment that the self-driving car industry was taking off and suddenly needed tons of workers, cheap labor.

18:18to do the annotation of self-driving cars, to teach self-driving cars how to navigate the road. And what I learned was that structurally, the data annotation industry has been designed to be exploitative. So she was working for Appen. And the way that Appen works is you can create an account and then when you log in, you have a stream of jobs that are available for you that are posted by companies that just tell you to do tasks that you don't know what they're for. And when she first joined the platform, this was generally a good premise. There were always tasks in her queue. She was able to take them.

18:58She was able to get paid, you know, several hundred dollars a month, which was enough for her to actually support herself, support her family in Colombia. But then there was an influx of workers and not enough jobs to go around. So by the time I met her, she was sometimes waiting four weeks at a time for a single task to appear. And that task could end up paying her just a few dollars. And the problem is she never knew when that task would arrive. And so there was one day that she was on a walk outside when a task suddenly arrived. And she sprinted back to her apartment to try and claim the task in time because these platforms pit workers against each other.

19:44So you have to claim it. And by the time she got back to her apartment, the task had already been claimed by someone else. It disappeared from her queue. And she decided at that point to never go on a walk outside ever again during the weekday. She learned that during the weekends, she could maybe get away with going out for a little bit. So she would limit herself to only a 30-minute walk on the weekends. It just made her life really small because she wasn't able to actually have freedom anymore. You know, I want to share an example. So at my company, Affectiva, we had a data annotation team based in Cairo, which is where I'm originally from.

20:25And they were mostly women. And I'm actually very proud of the team. They're still there, even after we solved the company. And as I was reading your examples of the stories of these data annotators, we actually decided to hire our data annotators as full-time employees. So they worked for the company. They had health care benefits. They had time shifts, right? So there is an alternative, more humane way of doing that. Absolutely. This is something that I also talk about in the book is that this could have actually been the primary opportunity for the industry to do what they long pay lip service to, which is the idea that they're going to redistribute the economic benefits that they concentrate within their hands.

21:07And if you think about it, you know, what better way to distribute economic benefits than to properly create these professionalized jobs for data annotation, which is a key component of the AI development supply chain, and make it into a dignified economic opportunity. And the elements that you described of what you ensured at Affectiva is also what researchers have long advocated for in the digital labor rights community, which is you give them full-time jobs, you give them benefits, you tell them who they're working for and why. You give them real managers that they can talk to so that if they are experiencing some kind of adverse effect from the job, they can actually contest that and raise awareness.

21:56But unfortunately, most companies choose not to follow that guidance at all. And it becomes a race to the bottom for how little you can get away with paying these workers. Yeah. There's also kind of the mental health side effects of some of these jobs, because, as you said, the input data to these LLMs are literally like everything and anything that's on the Internet. And I was struck by some of the stories of the data annotators looking through sexual content and violent content. Yeah, so Oskarina was pre-generative AI era, but then what's happened in the generative AI era is that there are still plenty of workers that are doing this work, but now the work has shifted to be problematic in and of itself.

22:43The content itself is dark and troubling. And so OpenAI at one point contracted these workers in Kenya to develop a content moderation filter. And I ended up going to speak with several of them. And I highlight the story of one man, Mo Fado Kine, who was on the sexual content team, where what he was expected to do was day in and day out read the worst sexual content on the Internet, as well as AI generated content on the Internet, where OpenAI was prompting its own models to imagine the worst content on the Internet, including sexual content. And he then had to categorize into a detailed taxonomy of is this, you know, sexual fantasies or is this sexual abuse?

23:25Is this sexual abuse that involves minors? And they all had a different tag of severity that he had to assign to them. And so MoFat ended up suffering the same fate that a lot of content monitors do in social media where his personality just fundamentally changed. Like he was extroverted and became very introverted and anxious. And when he would go home, he just could not engage anymore with his wife or with his stepdaughter who he loved and called his baby girl. And he also couldn't explain why, because he couldn't say to them, oh, my job involves reading sex content all day. That sounded really shameful.

24:08And so one day his wife asks him, I would like fish for dinner. So he goes to the store and buys three fish, one for him, one for her, one for the stepdaughter. And by the time he comes home, all their bags are packed and they're gone. And the wife just texts him, I don't understand the man you've become anymore. We're not coming back. I didn't put this in the book, but right after I came out of the interview, we walked into the hallway outside of his apartment and there was a neighbor's baby girl that was crawling around the hallway. And he just scooped up this little baby girl and was like doting on her and cooing to her and tossing her in the air.

24:47And I went back to my hotel and cried. I was like, this is I cannot believe that we are allowing these people to suffer in this way where he he lost his baby girl. And now, like, all you can do is play with his neighbor's baby girl. I mean, that's just so, so gut wrenching. It's so sad. You know, just in the news, just as we're having this interview, Scale AI announced they're getting a multibillion dollar investment from Meta. And Alexander Wang, the CEO, is joining Meta to lead AI efforts there. But OpenAI, which used Scale AI as a data provider, just announced that they're dropping the company as a data provider.

25:30Did you see that coming? I did not, but it makes a lot of sense on Meta's part to do something like that because Scale became the go-to platform for this kind of data annotation work. And it would make sense for a company to try and then buy it all up because Scale has visibility into the model development practices of all of the major players. purely as a business perspective, it is a very clever move to essentially, like they're not just acquiring the platform, they're acquiring the knowledge that that platform accumulated on all of the different contracts that they had. And of course, now the other companies feel that they can't use this service anymore because then it will just be a straight funnel of information directly to Meta.

26:27And so they're also taking one of the major players out of the market for other people to use. I mean, scale has also been riddled with labor exploitation. That is one aspect of the story that really hasn't been surfacing in the headlines of this major acquisition. AI can be extractive and exploitative, as Karen's reporting shows. And while data scraping and Unfair labor practices aren't often making headlines around AI. You know what is? The race towards the elusive goal of AGI. More on that in a minute. Stay with us.

27:18meet nicole nicholas capital one business customer and co-owner of anset uncles a plant-based restaurant and community space in brooklyn new york that got its start from a need for unity the inspiration it was born from the desire to create a space that felt like home where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew. It became overwhelming and we were like, we need home, but not in our actual home.

27:54We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step, but Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak as a small business. Finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful. You know, it just gave us that runway to be able to breathe a little bit.

28:28then you get to focus on the cooking of the food and making the experience great. To learn more, go to CapitalOne.com slash business cards.