Black Box: episode 1 – The connectionists

0:00This is The Guardian.

0:07Hey, it's Michael Safi, host of The Guardian Investigates, and I have some good news. We've made a brand new series of Black Box, our chart-topping podcast about the way AI is reshaping our lives. This time, it's all about AI chatbots. Gemini, Claude, ChatJPT. these new voices in our ear guiding what we think and feel and do. But should we trust them? Black Box, the chatbots, is coming very soon on the Guardian Investigates feed. And because Today in Focus is off this week, we're bringing you the first season of Black Box instead. These episodes are exactly as we broadcast them back in 2024. Black Box is next.

0:55The first time, Geoffrey Hinton ran into a scientific problem in the world that he couldn't explain. He was very young.

1:08So I remember when I was about five, coming home from school on a bus. It was an old bus with an engine that wasn't very well mounted, so the engine made the whole bus vibrate. And I was sitting on a seat that had some tough velvet-like material on it. And the seat sloped downwards towards the back. And I put an old British penny on the seat. Instead of sliding down the seat in the direction that gravity should have pulled it, the penny moved up. Like it was breaking the rules of the universe. and I couldn't believe it. So I kept putting the penny on the seat and different bits of the seat and it moved uphill.

1:57And I knew that was impossible. I didn't know why it was impossible but it obviously couldn't happen.

2:09If it were me on that bus, I'd have thought, that's weird. And then never thought about it again. But that's not Hinton. When you see something like that, when you see something that, according to your current model of the world, couldn't happen, some people say, oh, well, you know, the world's a complicated place, who knows? And other people say, I have to understand why that's going on. That shouldn't be. Why is that happening? I have to understand it. When he was much older, he finally cracked it. It was complicated, something to do with the fibres of the material on the bus seat. But whatever, the point is, he didn't stop thinking about it for like a decade.

2:49You were thinking about this for years after you were five years old until you figured out exactly how it worked. I had it in the back of my mind as a puzzle. It's something that had to be sorted out sometime. And I can't let go of things like that. I just get totally obsessed with things like that. And how the brain works with one of those things. Geoffrey Hinton is one of the pioneers of artificial intelligence. Geoffrey Hinton is known as a godfather. The godfather. The godfather of artificial intelligence. A technology that's being compared to the invention of railroads, of electricity, of the nuclear bomb.

3:29It's bigger than climate change. Way bigger than COVID. AI is one of the most profound things we are working on in humanity. And you and I are living through the unique moment when it collides with humanity for the first time. It's the most existential debate and challenge humanity will ever face.

3:52This is a series about that collision. Starting with the story of the man who did maybe more than anybody to get us here. who spent his entire career on the fringes, working on something most scientists thought was a fantasy until the world sat up and realised it was going to change everything.

4:13From The Guardian, I'm Michael Safi. This is Black Box, episode one, The Connectionists.

4:25That's the subject of our program today, artificial intelligence. Artificial intelligence is transforming our world. Intelligent creatures on this planet that are not human. It can create almost anything from coding video games to passing standardized tests. And it can generate full photorealistic or stylized image or whatever. It is smarter than you are. And the race for domination has begun.

4:56To understand artificial intelligence, you need to start with the human brain. That's what Geoffrey Hinton was really interested in as a kid growing up in England in the 1960s. By the time he had finished school, scientists had figured out nuclear fission, the structure of DNA. They were sending astronauts into outer space. But when it came to understanding the brain, the thing that made all these incredible things possible, we still only understood the basics. Here's the basic version of the basics. The brain is made up of these cells called neurons. And when we think or move or do pretty much anything, those neurons start firing like they're talking to each other.

5:42but what they're saying and how all that chatter leads to ideas and memory and learning, that was a mystery. Scientists have a name for this kind of thing, where you see what goes in and what comes out, but have no idea what's happening in the middle. They call it a black box. And like with the coin on the bus seat that moved upwards, A young Geoffrey Hinton just couldn't move past that. How could we not know? He had to work it out.

6:18After school, he applied to Cambridge. And they accepted me. I think they mainly accepted me because my father had been there. So I think it was, I think I was benefiting from white privilege. Hinton treated Cambridge like a hotel buffet. He tried a bit of philosophy, didn't like it. He was a student of architecture for a single day and then dropped it. He studied chemistry, not for him. He knew the question he wanted to answer. He just couldn't find the path that would get him there. He eventually graduated with a degree in experimental psychology and left university behind. He got another job.

6:56I became a carpenter. A carpenter? Yes. I wasn't a very good carpenter. In some of the pictures from around this time, Hinton looks younger than his age, which now at 76, he still does. He has the same faint trace of a smile and this single tuft of hair that's sprung loose and runs down his forehead. In the pictures, it's still brown, but when we talk from his house in Canada, it's gone white. It was in this period after university, a kind of intellectual exile, that Hinton encountered a relatively new idea. Like I was saying, the brain is full of these neurons that are constantly pinging with electrical signals, like they're talking to each other.

7:39But this new theory said, when those neurons fire at the same time, something really interesting is happening. As they fire together, they wire together. And the more these neurons fire at the same time, the stronger the connection between them becomes. It goes from being a dirt track to a paved road to eventually a major highway. And this process of the connections getting stronger, that's what learning looks like.

8:14One example. Say you're walking home and an angry dog moves into a house on the next street. In your brain, the neurons that are associated with this street and the neurons that are associated with fear start to fire at the same time. And they start to wire together. You learn to fear this street. Equally, if the dog moves away, those neurons don't fire together anymore and the connection between them withers. And eventually, the feeling of fear you get every time you're on that street, it fades away. And Hinton felt like, this theory, this is what he'd been looking for. So on the weekends, he'd go to the library, not far from the Guardian office here in London, and start sketching out ideas.

9:00Yes, while I was a comment, I used to go to the Islington Public Library every Saturday morning and write notes in my notebook about how the brain might work. And he thought one way to test this idea of how the brain learned would be to build it artificially. I decided to apply to Edinburgh to do artificial intelligence. I thought artificial intelligence would be the way forward. Okay, so Geoffrey Hinton is not the first person to have this idea, to build a computer that's modelled on the brain. And just for a minute, I want to tell you about one of those earlier attempts, just to give you a sense of what Hinton was up against.

9:44In the 1950s, an American scientist named Frank Rosenblatt built a computer that he called the Perceptron.

9:57Michael Safi:And I love that name. That's such a science fiction name for something, the Perceptron. This is Michael Waldridge, a computer science professor at Oxford. The Perceptron was the first version of the type of AI that we have today, an artificial neural network. Like the brain, it had its own neurons, connected by wires going in a million directions. And for its time, it was amazing. This perceptron is being trained to recognise the difference between males and females. It is something that all of us can do easily, but few of us can explain how. If you showed it enough examples, it could eventually teach itself to identify numbers, tell left from right, or tell the difference between a dog and a cat.

10:37After training on lots of examples, It's given new faces it has never seen and is able to successfully distinguish male from female. It has learned.

10:49Michael Safi:And it was a complete sensation. His work gained a lot of attention in the 1960s and there was a common phrase at the time to describe these things. Imagine if you can an electronic brain operating at millionth of a second speed. People called them electronic brains. I say brain because the new electronic central office will almost blink for itself. Journalists came from all over America to interview Rosenblatt. The hype over what these things would one day be able to do was huge and kind of familiar. What really worries me today is what's going to happen to us if machines can think. But the sad truth is he had extremely inflated expectations.

11:32Michael Safi:and he made, as is the tradition of AI researchers, some overblown predictions about what it was going to be capable of. And all that buzz around AI, the overselling, the failure to live up to expectations, it led to a backlash.

11:51Good evening and welcome to the Royal Institution. Around the time that Hinton was getting into this stuff, there was a debate about AI on British TV. In 1971, the people who hand out research funding asked a maths professor called Sir James Lighthill to look into the field of artificial intelligence. So Lighthill goes on TV to present his findings. The room is packed with some of the most important scientists in the country. There may be some people who try to make us think we can see that old general purpose robot shimmering there on the horizon, but he's a mirage. And Lighthill is brutal. He basically says, AI doesn't work.

12:35It's a waste of time and money. One can feel dubious to a very high degree about predictions of a general purpose robot. Well, perhaps this is a good time to leave.

12:47Michael Safi:And this was the beginning of a period where artificial intelligence was kind of treated like homeopathic medicine. You know, something on the fringes, something with, you know, a questionable scientific basis. And it remained there for really, really a long time. This period came to be known as the AI winter.

13:12And that was the world Geoffrey Hinton was entering when he hung up his carpenter's tools and found his way back to university, enrolling this time on one of the few remaining AI courses. an overhyped idea, the stuff of science fiction. But Hinton didn't care. Do you remember, though, when the report came out, the kind of effect that it had on this world that you had started to steep yourself in? It was fairly devastating, particularly for employment opportunities. Basically, the funding dried up, and it made life particularly tough for anybody who wanted to get an academic position in Britain doing AI.

13:50So by the time I got my PhD, there just weren't any academic positions in AI. And at the time, do you remember how people around you spoke of and thought of neural networks and their potential? Yes, I do. There was universal agreement that they were rubbish. They told me I had to stop it and do something sensible, stop doing this silly stuff. And I wanted to carry on doing this silly stuff. I applied for jobs in England and I couldn't get an interview for a job so I saw an advertisement for a job in California and there life was quite different there there were people who believed there was potential in neural networks and so I was with people who didn't think what I was doing was nonsense for the sort of first time and that was wonderful the modern perceptrons are called neural networks and the people who build them are a growing movement called connectionists.

14:52In California, Hinton joined a small fringe group of psychologists, philosophers, computer scientists who still believed, working quietly for years under the radar. Connectionists are seeing how much they can get their neural networks to learn. They called themselves the Connectionists, inspired by the idea that first grabbed Hinton back in the 1970s that the secret to learning was in the way that neurons in the brain made connections and believing that maybe you could build computers the same way. Not programming them, writing rules line by line, but building a network of artificial neurons, feeding it information, letting those neurons fire, form their own connections, and as those connections form, watching it learn.

15:46Pretty soon, they cleared their first big hurdle. The problem with the perceptron was that it was inspired by the brain, but in a really key way, it was nothing like it. In the brain, the neurons are structured in these layers, like a lasagna, and this design is one of those strokes of genius that you see in nature. It seems simple, but layers just make the whole process so much more efficient and powerful. But trying to recreate that structure in a computer program, for a long time the consensus was it was mathematically impossible. Couldn't be done. Except, throughout the 1970s and 80s, at a few different institutes around the world, a handful of computer scientists, including Geoffrey Hinton, Crackton, had to build a neural network in layers.

16:35Nobody outside this tiny world really noticed at the time. But now we can look back and see they had fired a starting gun.

16:48How important was that breakthrough? It was extremely important. And it really opened up all of the abilities that we see in neural networks today. That's Melanie Mitchell. Back then, she was a young computer science student. Now she's a professor at the Santa Fe Institute. And when she was starting out, she was told these neural networks would never go anywhere. But she also remembers watching how, after this breakthrough, the field started to change.

17:21There was like a very early project that was called NetTalk. I like to go to my grandmother's boss. Well, because she gave us candy. which was a neural network that could take text and turn it into speech. It wasn't very good at all compared to what we have today, but it learned that just from data. There were no rules programmed into it, and so that was seen as a really interesting kind of advance in the neural network world. He won't stop jumping or running in the bathtub. In the bathtub? No. In the crib. OK, so it wasn't pretty. But it was progress. The neocognitron is a multilayered neural network consisting of a cascade of many layers of simplified neural cells.

18:13In the same year, researchers came up with a way to dramatically improve how neural networks see. Let's draw a numeral 2 on the tablet. From the analysis of the stimulus features, the input pattern is recognised as 2. It would eventually lead to things like being able to unlock your iPhone using your face, to facial recognition software, being able to track you as you walk across the city. But the implications of what they were doing, what this technology might one day mean for privacy, jobs, elections, fake news, back then, it was hard to imagine. We were so far off from being able to produce things that worked well that we didn't worry about that kind of thing.

18:57And until quite recently, I didn't worry much about those kinds of things. The reason Hinton didn't worry was that back then, the machines were still missing two really big things. First was, to build a neural network big enough to rival the brain, you'd need huge amounts of computing power. And back then, that didn't really exist. Second, it would need a vast amount of training material to learn from. And where would you ever find trillions of human conversations and messages and pictures all sitting there downloadable in some kind of interconnected network of computers worldwide? It would never happen, right?

19:40Imagine a world where every word ever written, every picture ever painted and every film ever shot could be viewed instantly in your home via an information superhighway. But throughout the 90s and the 2000s, Computer power was growing. Purchasing an airline ticket from home. Checking out your bank account. Billions of people were going online. And I found this satellite weather map in there. Uploading thousands of terabytes of information every second. You can get news, recipes. This digital mirror of humanity. The best of us and the worst. Kids that are called super predators. No conscience.

20:18And after seven decades on the fringes, neural networks, they were starting to break through. Is this real life?

20:30Sometimes, with big breakthroughs, there's a precise moment where you can put your finger on where an idea goes from the fringes of science, from labs and academic journals, to the real world. For AI, that moment came in 2012. Okay, that was quite interesting. There's this annual competition for computer scientists called ImageNet. And it's like the picture round at a pub quiz, but instead of you and your mates trying to guess the celebrity, it's specially designed computer programs trying to identify what they see in thousands of random pictures. Cats, dogs, specific kinds of flowers, whatever.

21:06And whoever's computer gets the most right answers, wins. Earlier ImageNet winners had been improving their systems by like a percentage point a year. Just this slow grinding progress. But Geoffrey Hinton and two of his grad students, Ilya Sutskiva and Alex Krzyzewski, figured a neural network could do this much better. Ilya was convinced that the techniques we had would actually win this competition and he really insisted we work on it. So early in 2012, the three of them started building one. Alex was an incredibly good programmer. He's probably one of the best two programmers I know. In his bedroom at his parents' house in Toronto, Alex spent weeks feeding the network images, watching it make connections in the data, identify patterns, teach itself to distinguish between a Doberman and a Dachshund, a Sunflower and a Snapdragon.

22:02Alex was pouring hours into it and starting to neglect his work for uni. He actually had an assignment overdue for one of Hinton's classes. And he kept putting it off. So I made a deal with him that every time he improved the score on ImageNet by 1%, he could put it off for another week. And he managed to put it off for many, many weeks. So in the process of avoiding having to write this paper, he did what was probably the most influential piece of research this century.