What the Head of the New UK AI Foundation Model Taskforce Thinks of AI

0:00Today on the AI Breakdown, we long read a set of different excerpts from pieces from Ian Hogarth, the recently appointed chairman of the UK Foundation Model Task Force. The AI Breakdown is a daily podcast and video about the most important news and stories in AI. Like, subscribe, and share, and go to breakdown.network for more information. Hey, hello, friends. Welcome back to the AI Breakdown. Today, we're doing something a little bit different with our long reads. Earlier this week, entrepreneur and investor Ian Hogarth was named the chairman of the UK Foundation Model Task Force. This is Rishi Sunak's 100 million pound task force that's meant to help the UK become a hub and a leader when it comes to AI regulation and policy.

0:42Now, Ian is a really dynamic and interesting choice, someone who comes from the tech and entrepreneurial world, but who has spent the last few years getting deeper and deeper into AI and thinking about it also from a policy perspective. So what we're going to do today is read some excerpts from a number of pieces that show his evolving thinking on AI. The first is from a piece he called AI Nationalism in 2018. The second is from a more recent piece, We Must Slow Down the Race to God-like AI that appeared in the Financial Times. And the third is a thread and an op-ed around the announcement that he had become the chairman of the UK Foundation Model Task Force.

1:16So let's first go back to 2018. On June 13th, Ian wrote, For the past nine months, I have been presenting versions of this talk to AI researchers, investors, politicians, and policymakers. I felt it was time to share these ideas with a wider audience. The central prediction I want to make and defend in this post is that continued rapid progress in machine learning will drive the emergence of a new kind of geopolitics. I have been calling it AI nationalism. Machine learning is an omni-use technology that will come to touch all sectors and parts of society. The transformation of both the economy and the military by machine learning will create instability at the national and international level, forcing governments to act.

1:54AI policy will become the single most important area of government policy. An accelerated arms race will emerge between key countries and we will see increased protectionist state action to support national champions, block takeovers by foreign firms, and attract talent. This arms race will potentially speed up the pace of AI development and shorten the timescale for getting to AGI. Nationalism is a dangerous path, particularly when the international order and international norms will be in flux as a result, and in the concluding section I discuss how a period of AI nationalism might transition to one of global cooperation where AI is treated as a global public good.

2:28Now, section two we're going to skip over. It's called Progress in Machine Learning, and in it Ian points to a number of different recent updates that include Microsoft achieving human parity on Mandarin and English translation, DeepMind building AlphaGo, and then AlphaZero, which beat WorldGo champions and later top-performing chess computers. Section three is called Three Forms of Instability. Ian writes, So why does this matter to nation-states? There are three main ways in which accelerating progress in machine learning could create instability in the international order. Commercial applications of machine learning will create vast new businesses and destroy millions of jobs.

3:03In the extreme case, the country that invests the most efficiently may end up the strongest economically. Machine learning will enable new modes of warfare, both sophisticated cyber offense and defense capabilities, but also various forms of autonomous and semi-autonomous weaponry. In the most extreme case, the country that invests the earliest and most aggressively may end up in a position of military supremacy. Eventually, more general purpose AI will enable a fundamental speed-up in science and technology research. In my opinion, this might actually be the most profound source of instability.

3:31Consider, for example, the state whose leadership in AI enables them to be the first to develop a viable fusion reactor for power generation. Again, in the extreme case, this might enable a country to achieve Wakandan technological supremacy. Machine learning, to use Jack Clark's term, is a uniquely omni-use technology that could impact almost every area of national policy. Human intelligence has shaped everything we see around us, so our ability to build machines with greater and greater intelligence could eventually have the same impact. Ambitious governments have already started to see machine learning as the core differentiating technology of the 21st century, and a race has already commenced.

4:04This race will come to bear some similarity to the nuclear arms race of the last century and geopolitical tensions and alliances between nation-states and multinational companies over oil. In Section 4, Ian discusses industry mix, labor costs, demographics, graphics, basically talking about how while AI impact will have some common threads throughout the world, the specific impacts will also vary country to country based on things like what types of industries they have and their approaches to welfare and redistribution. In Section 5, Ian identifies that there is a blurring line between public and private sectors.

4:34Ian writes, there are incredibly powerful non-state actors who are also competing furiously to develop this technology. All of the seven most important technology companies in the world, Google, Apple, Amazon, Facebook, Alibaba, Tencent, and Baidu are making huge investments in AI, from low-level frameworks and silicon to consumer products. It goes without saying that their expertise in machine learning leads any state actor at the moment. As the applications of machine learning grow, the interactions between these companies and different nation-states will grow in complexity. States have historically played a crucial role in underwriting long-term, high-risk research in science and technology by funding either academic research or the military.

5:10These technologies are often then commercialized by private companies. With the rise of visionary and wealthy technology companies like Google, we are seeing more high-risk long-term research being funded by the private sector. This creates tensions when the interests of a private company like Google and a state are not aligned. In Section 6, Ian identified that China in 2018 was way out ahead. Indeed, he wrote in developing a national strategy for AI, China is way out ahead of everyone else. For China, over the past couple decades, protectionism has been a winning strategy in developing enduring domestic technology companies, and it has ultimately enabled China to be the only other country in the world with AI companies to rival Americas.

5:45Now the piece, which is awesomely long and thorough, then goes through key events in the arms race so far, AI nationalism policies, what can countries that aren't China or America do, the strange case of the UK, rogue actors and how they complicate this whole mix, and then ultimately comes to his concluding section called Engineers Without borders. In it, Ian gives his widest view. He says, Personally, I believe that AI should become a global public good, like GPS, HTTP, TCPIP, or the English language. And the best long-term structure for bringing this to fruition is a non-profit, global organization with governance mechanisms that reflect the interests of all countries and people.

6:24The best shorthand I have for this is some kind of cross between Wikipedia and the UN. While the idea of AI as a public good provides me personally with a true north, I think it is naive to hope we can make a giant leap there today, given the vested interest and misaligned incentives of nation-states, for-profit technology companies, and the weakness of international institutions. I believe that we are likely to go through a period of AI nationalism before we get to a place where AI is treated like a public good. And that, to use Orwell's distinction, a kind of AI patriotism is likely to be a good thing for smaller countries in the short term.

6:55Taking the example of the UK, I am in favor of a more expansive national AI strategy to protect the UK's economic, military, and technological interests, and to give the UK a credible seat at the table when global issues around AI are being worked out. That will help ensure that the UK's economic interests and values are considered. I believe it is necessary for the UK government to take steps towards investing in and protecting its homegrown AI companies and institutions to allow them to play a larger role on the world stage, independent of America and China. I have lived in both America and China, and during that time developed enormous respect and affection for both of those countries.

7:26That does not prevent me from believing the UK should protect the economic interests of its citizens, and I would like to see the UK play a material role in shaping the future of AI. So that is Ian speaking in 2018. Fast forward to April 13th of this year, the day after this podcast first came out. On that day, Ian wrote an extremely extensive piece for the Financial Times called We Must Slow Down the Race to Godlike AI. Ian starts that piece, On a cold evening in February, I attended a dinner party at the home of an artificial intelligence researcher in London, along with a small group of experts in the field.

8:00He lives in a penthouse apartment at the top of a modern tower block, with floor-to-ceiling windows overlooking the city's skyscrapers and a railway terminus from the 19th century. Despite the prime location, the host lives simply, and the flat is somewhat austere. During dinner, the group discussed significant new breakthroughs, such as OpenAI's ChatGPT and DeepMind's Gato, and the rate at which billions of dollars have recently poured into AI. I asked one of the guests who has made important contributions to the industry the question that often comes up at this type of gathering, How far away are we from artificial general intelligence?

8:28AGI can be defined in many ways, but usually refers to a computer system capable of generating new scientific knowledge and performing any task that humans can. Most experts view the arrival of AGI as a historical and technological turning point, akin to the splitting of the atom or the invention of the printing press. The important question has always been, how far away in the future this development might be. The AI researcher did not have to consider it for long. He replied, it's possible from now onwards. This is not a universal view. Estimates range from a decade to a half century or more.

8:58What is certain is that creating AGI is the explicit aim of the leading AI companies, and they are moving towards it more swiftly than anyone expected. As everyone at the dinner understood, this development would bring significant risks for the future of the human race. If you think we could be close to something potentially so dangerous, I said to the researcher, shouldn't you warn people about what's happening? He was clearly grappling with the responsibility he faced, but, like many in the field, seemed pulled along by the rapidity of progress. When I got home, I thought about my four-year-old, who would wake up in a few hours.

9:27As I considered the world he might grow up in, I gradually shifted from shock to anger. It felt deeply wrong that consequential decisions potentially affecting every life on Earth could be made by a small group of private citizens without democratic oversight. Did the people racing to build the first real AGI have a plan to slow down and let the rest of the world have a say in what they were doing? Now, next in this piece, Ian talks about his background. And the important thing to note is that this is not someone who has been in academia or policy all his life. He's a tech guy. He's always worked in tech.

9:54He sold a company. He's backed more than 50 AI startups. So he's not coming at this from a Luddite perspective. And still, as Ian writes at that dinner in February, significant concerns that my work has raised in the past few years solidified into something unexpected, deep fear. Now, continuing to excerpt from the piece, Ian writes, a three-letter acronym doesn't capture the enormity of what AGI would represent. So I will refer to it as what it is, godlike AI. a super-intelligent computer that learns and develops autonomously, that understands its environment without the need for supervision, and that can transform the world around it.

10:28To be clear, we are not here yet. But the nature of the technology means it is exceptionally difficult to predict exactly when we will get there. Godlike AI could be a force beyond our control or understanding, and one that could usher in the obsolescence or destruction of the human race. Recently, the contest between a few companies to create godlike AI has rapidly accelerated. They do not yet know how to pursue their aims safely and have no oversight. They are running towards a finish line without an understanding of what lies on the other side. The current era has been defined by competition between two companies, DeepMind and OpenAI.

10:58They are something like the jobs and gates of our time. Now, he then goes on to talk about the forming and foundation of both DeepMind and its acquisition by Google, as well as OpenAI and its conversion from a non-profit to a for-profit, and talks about how much of the emphasis of both of these companies has been on the application of AI, or the exploration at least of AI, in areas like gaming and chatbots. Ian writes,

11:48Earlier this year, Altman said, quote, the bad case, and I think this is important to say, is like lights out for all of us. Why are these organizations racing to create Godlike AI, Ian continues, if there are potentially catastrophic risks? Based on conversations I've had with many industry leaders and their public statements, there seem to be three key motives. They genuinely believe success would be hugely positive for humanity. They have persuaded themselves that if their organization is the one in control of Godlike AI, the result will be better for all. And finally, posterity. The individuals who are at the frontier of AI today are gifted.

12:20I know many of them personally. But part of the problem is that such talented people are competing rather than collaborating. Privately, many admit that they have not yet established a way to slow down and coordinate. I believe that many would sincerely welcome governments stepping in. For now, the AI race is being driven by money. Since last November, when ChatGPT became widely available, a huge wave of capital and talent has shifted towards AGI research. We've gone from one AGI startup, DeepMind, receiving$23 million in funding in 2012, to at least eight organizations raising$20 billion of investment cumulatively in 2023.

12:50Now Ian then returns to some of the themes from his AI nationalism piece talking about the geopolitical dimension of this, and finally concludes with a quick discussion of alignment. Alignment, Ian writes, is essentially an unsolved research problem. We don't yet understand how human brains work, so the challenge of understanding how emergent AI brains work will be monumental. When writing traditional software, we have an explicit understanding of how and why the inputs relate to outputs. These large AI systems are quite different. We don't really program them, we grow them. What is more concerning is that the number of people working on AI alignment research is vanishingly small.

13:22For the 2021 State of AI report, our research found that fewer than 100 researchers were employed in this area across the core AGI labs. As a percentage of headcount, the allocation of resources was low. DeepMind had just 2 % of its total headcount allocated to AI alignment. OpenAI had about 7%. The majority of resources were going towards making AI more capable, not safer. We have made very little progress on AI alignment, in other words, and what we have done is mostly cosmetic. Now, Ian concluded that piece with a set of very loose recommendations that basically amounted to governments need to get more involved, and people who want to have a say in this need to start getting louder.

13:57Well, as of last Monday, he will now have a chance to help lead that conversation. On June 18th, he tweeted, I'm honored to be appointed as the chair of the UK's AI Foundation Model Task Force. I wrote in 2018 about how accelerating AI progress would create new geopolitical challenges. In April 23, I wrote an essay for the FT, We Must Slow Down the Race for Godlike AI, that highlighted the risks of a small number of AI companies racing to create ever more capabilities without enough progress on AI safety or regulation. The world has significantly shifted since then. Pioneers in the field like Jeffrey Hinton have spoken out.

14:29An open letter signed by a broad coalition of AI experts compared the risks to nuclear weapons or pandemics. And at a pivotal moment, Rishi Sunak has stepped up and is playing a global leadership role. He has pledged 100 million pounds on AI safety, the largest amount ever committed to this field by a nation state. The field of AI safety has been significantly under-resourced even as funding for AGI companies has now crossed a cumulative$20 billion. We have 100 million pounds to spend on AI safety and the first global conference to prepare for. I want to hear from you and how you think you can help.

14:57The time is now and we need more people to step up and help. I am fundamentally optimistic about the potential for science and technology to transform our lives for the better. The opportunity for AI to be a force for good are truly remarkable, but we need to do it safely. Ian then echoed those themes in an op-ed for The Times. Now, I'm enthusiastic about this appointment. I think Ian's a great choice, and I really like people who have this perspective of a base level of techno-optimism also coming into this murkier, newer space and being able to rewrite some of their priors and look at it from first principles.

15:28I think that to achieve what we want to achieve here, it's going to take people who can translate society, policy, and industry. And I think, frankly, that for industry to respect the policy side, to some extent, it's going to have to speak to them in their own language. I also think that Ian is right to recognize that even in just the two months since he wrote that piece, the landscape for the AI safety conversation looks very different. To me, it's pretty obvious why. The hundreds of millions of people, if not billions of people, who have learned about this space because they tried a generative AI tool such as ChatCpt or MidJourney or something like that, feel to me, frankly, much more comfortable with the idea that these technologies could get out of our hands.

16:11When you've just experienced something that feels like magic for the first time, it's not hard to imagine that that magic might metastasize in ways that you can't understand or see now. I think this is borne out in people's response to Jeffrey Hinton and Yahshua Bengio and all of this new media attention on AI risk. But I also think that people are going to want to move the conversation to what we actually do. The six-month pause idea not only didn't hit people as the right approach, the most common response even for people who thought it might be a good idea in the short term was, and then what?

16:42But the ground is fertile for the conversation, and I'm glad that we're actually having it. That is it for today's Long Read Sunday. I hope this was helpful. I hope this introduced you to Ian and how he's thinking about governments in the UK and AI safety in general, check out his Twitter at Soundboy, and he has a bunch of links pinned to his profile page for where you can fill out a form to see if you can be involved with the Foundation Model Task Force. If you're enjoying the AI Breakdown, please like, subscribe, and share it. Go check out the YouTube if you're listening, go check out the podcast if you're watching, and until next time, peace.

17:25Thank you.