Are We At Peak AI?

0:00This is an iHeart Podcast.

0:30Thomson Reuters and specialized bikes have since they upgraded to the next generation of the cloud. Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? Better? OCI costs up to 50 % less for computing, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds.

1:12This is the cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash strategic. That's oracle.com slash strategic. The secret to steady business growth? Fast funding with Amerifactors. If you're a business owner in need of capital for payroll, inventory, or expansion, you need Amerifactors. Get the help you need even with less than perfect credit, including bankruptcies. Amerifactors provides ready cash flow through accounts receivable management. Get tailored solutions. Call today for a free, no obligation, no impact to your credit score quote at 800-884-FUND.

1:51That's 800-884-3863. Or visit amerifactors.com.

2:01Hello and welcome to Better Offline. I'm your host, Ed Zitron.

2:16It's been just under a year and a half since JetGPT, an AI-powered chatbot launched by so-called non-profit OpenAI, ushered in a new investor and media hype cycle around how AI would change the world. ChatGPT's instant success made both OpenAI and its CEO, Sam Altman, overnight celebrities as a result of ChatGPT's alleged intelligence, which seemingly allowed you to do everything from generate an entire essay from a simple prompt to writing entire reams of software code. You can theoretically ask it anything, and it will spit out an intelligent-sounding response thanks to being trained on terabytes of text data, like a search engine that's able to think for itself.

2:56Eh, big problem is, ChatGPT doesn't think at all. It doesn't know anything. ChatGPT is actually probabilistic. It uses a statistical model to generate the next piece of information in a sequence. If you ask it what an elephant is, it'll guess that the most likely answer to that prompt is that an elephant is a large mammal and then perhaps describe its features, such as a long trunk. ChatGPT doesn't know what an elephant is, or what a trunk is, or what the elephant-hunt-i family is. It's simply ingested enough information to reliably guess what an elephant is meant to be, or indeed, know that that's what you're asking it.

3:32This is the technology underpinning the latest artificial intelligence boom. It's called generative artificial intelligence, and it's powered by large language models. And they underpin tools like OpenAI's ChatGPT, Anthropix Claude, X.com's horrifying chatbot Grok, and of course, Google's Gemini. Essentially, they're AI systems that ingest vast quantities of written text or other data, and then through mathematics, try and identify patterns of relationships between words or symbols, or basically any meaning from the text or thing they're being fed. And it almost seems like magic, because it's able to generate this plausible-seeming, almost-human content at this remarkable speed.

4:15These models are now capable of generating text, images, and even video in response to simple chat prompts, all by learning the patterns and structures of their data. Yet underneath the hood, there's always something a bit wrong. Generative AI at times authoritatively spits out incorrect information, which can range from funny, like telling you that you can melt an egg, to outright dangerous, like when the recently launched AI-powered New York City chatbot for small business owners started telling them that it was legal to fire somebody for refusing to cut their dreadlocks. This is why you'll see strange glitches in images generated by AI, hands with too many fingers, horrifying looking people in the back of realistic looking photos, and so on and so forth.

4:56Because these models don't actually know what anything is. They don't have meaning. They don't have consciousness or intelligence. They're guessing. And when they guess, they sometimes hallucinate, which I'll get to soon. And while they might be really, really good at guessing, they're effectively a very, very powerful version of autocomplete. I don't know anything. I really mean that. These things aren't even intelligent. But because these models seem like they know stuff, and they seem to be able to do stuff, and the things that they create almost seem right, the media and the vocal investor class on Twitter have declared that large language models would change everything.

5:38To them, LLMs like ChatGPT would upend entire business models, render once unassailable tech giants vulnerable, and rewrite our entire economic playbook by turning entire industries into something you'd tell a chatbot to do in a sentence. You know, it doesn't really matter that generative AI is mostly good at pumping out reams of generic slop, and that it's also clogging services like Amazon's Kindle eBook store, and I guess the rest of the internet with generative content, that doesn't matter at all. Because it's kind of good. Obviously, I'm being sarcastic. This is all very, very bad. In the last year, there have been hundreds of mewling articles about how AI will replace everything from drive-thru workers to medical professionals.

6:21Theoretically, you could just feed whatever information a potential customer could ask for into a vast database and have an AI chew it up and then they could just generate exactly the answer you'd need. AI would just naturally slip into areas of disorganization and inefficiency and spit out remarkable new ideas, all with minimum human input. In every one of these stories carries with them a shared belief. One might even call it a shared hallucination. And they all believe that generative AI will actually be able to do these things, that it'll actually be able to replace people. And what we're seeing today is just the beginning of our glorious automated future.

7:00But what if it's not? What if generative AI can't actually do much more than it can today? What if we're actually at peak AI? In the next two episodes, I'm going to tell you why I think that is. I'm going to tell you how I think this whole thing falls apart.

7:40so i've shopped with quints before they were an advertiser and after they became one and then again before i had to record this ad i really like them my green over shirt in particular looks great I use it like a jacket. It's breathable and comfortable and hangs on my body nicely. I get a lot of compliments. I liked it so much. I got it in all the different colors, along with one of their corduroy ones, which I think I pull off. And really, that's the only person that matters. I also really love their linen shirts, too. They're comfortable, they're breathable, and they look nice. Get a lot of compliments there, too.

8:08I have a few of them. Love their rust-colored ones as well. And in general, I really like quints. The shirts fit nicely, and the rest of their clothes do, too. They ship quickly. They look good. They're high quality, and they partner directly with ethical factories and skip the middleman. So you get top tier fabrics and craftsmanship at half the price of similar brands. And I'm probably going to buy more from them very, very soon. Keep it classic and cool this fall with long lasting staples from Quince. Go to quince.com slash better for free shipping on your order and 365 day returns. That's Q-U-I-N-C-E dot com slash better.

8:41Free shipping and 365 day returns. Quince.com slash better. Are you a business owner or entrepreneur in need of a fast funding solution? Amerifactors provides fast capital. Call today for a free no obligation quote at 800-884-FUND. Startups, rapid growth stage or struggling, even with less than perfect credit, including bankruptcies. In business for over 34 years, thousands of satisfied clients have transformed their cash flow with Amerifactors. Call Amerifactors today at 800-884-FUND. That's 800-884-3863. or visit amerifactors.com. AI's media hype train has been fairly relentless since November 2022 when ChatGPT launched.

9:26And AI champions like OpenAI CEO Sam Altman have proven all too willing to grease its wheels, making bold promises about what AI could do and how today's problems are so easily overcome. It's also helped that the tech media has largely accepted these promises without asking basic questions like how and when will it do this stuff? Can it do this stuff? Don't believe me? Go and look at any interview with Sam Altman from the last few years. Watch any of them. In fact, just look at any AI figurehead getting interviewed and count the amount of times they've actually received any pushback or been asked to elaborate on any specific issue.

10:04It's actually very rare. Let me play you one of the few times that anyone's actually interrogated an AI person, specifically Joanna Stern of the Wall Street Journal, who you might remember from the Vision Pro episode of Better Offline, she interviewed OpenAI's chief technology officer, Mira Marati, about Sora, which is OpenAI's video-based version of ChatGPT, where you can theoretically ask it to generate videos. Just to be clear, it's unreleased and unclear whether it'll ever actually get released, and the videos look good at first, then they look really weird. But just listen to this particular question.

10:36It's Joanna asking Mira, the CTO of OpenAI, an$80 billion AI company, hey, did you train on YouTube? What data was used to train Sora? We used publicly available data and licensed data. So videos on YouTube? Now I encourage you to go and look up this clip because at this point, Mirati makes the strangest face I've ever seen in a tech interview. I'm actually not sure about that. Okay. Videos from Facebook, Instagram? You know, if they were publicly available to use, there might be the data, but I'm not sure. I'm not confident about it. What about Shutterstock? I know you guys have a deal with them.

11:31I'm just not going to go into the details of the data that was used, but it was publicly available or licensed data. The remarkable part about this interview is that it's a relatively simple question. You, as the CTO of an$80 billion AI company, what training data did you use to train your model? Did you use YouTube? It's a yes or no question, Mira. Mira, answer the bloody question, Mira. All right, right. The answer is, of course, OpenAI likely trained its video generating model Sora on YouTube videos, which might be why they're yet to launch it. And the videos generated by Sora also feature some remarkably similar images to, say, SpongeBob SquarePants, and I wouldn't be surprised if they carry with them multiple weird biases about race and gender that we'll see in the future.

12:24But also, when you watch these videos, much like most generative AI content, there's something a bit off about them. In the Wall Street Journal's interview, you get to see some of the prompts that were used and some of the videos that came out, and you see crazy things happening, like a robot completely changing shape as it turns, cars disappearing and appearing behind the robot. It's not very good. It seems cool at first. If you squint really hard, it looks real, but there's always something off. And that's because, as I've said before, these models don't know anything. They don't know what a robot looks.

12:56They can make a really good guess, though. Anyway, Stern's interview with Marathi of OpenAI is a great example of how the entire AI artifice falls apart at the slightest touch because it's fundamentally flawed and not actually able to deliver the society-defining promises that Sam Altman and the venture capital sect would have you believe. In a year and a half, despite billions of dollars of investment, despite every major media outlet claiming otherwise, generative artificial intelligence has proven itself incapable of replacing or even meaningfully enhancing human work. And the thing is, all of these problems I'm talking about with generative AI, all of these hallucinations, all of these weird artifacts that are popping up throughout these videos, the weird mistakes that the texts that are popped out by ChatGPT have, all of these problems are problems that aren't necessarily just technological.

13:51They're physics. They're mathematics. these aren't things you can just outrun and i believe that there are four intractable problems that will stop generative ai from progressing much further than it is today the first is of course its energy demands the massive amounts of power it requires the second are its computational demands the amount of compute power it requires to even crunch the simplest things out of chat gpt its hallucinations, the authoritative failures it makes when it spits out nonsense or creates a human hand with 18 fingers, and of course the fact that these large language models have an insatiable hunger for more training data.

14:35Now let me break that down. Large language models are extremely technologically and environmentally demanding. The New Yorker reported in March 2024 that ChatGPT uses more than half a million kilowatt hours of electricity to respond to the 200 million requests it receives in a day, or 17 ,000 times the amount that the average American household uses in a day, and others have suggested it might be as high as 33 ,000 households worth. Generative AI models demand specialist chips called graphics processing units, typically a souped-up version of the technology used to drive the graphics in a gaming console, albeit at a much higher cost.

15:14with each one costing tens of thousands of dollars each. They do this because large language models like ChatGPT are highly computationally intensive. I'm going to break that down, don't worry. When you ask ChatGPT a question, it tokenizes it, breaking it down into smaller parts for the model to understand. It then feeds these tokens into various mechanisms that help it understand the meaning of the thing you asked it to do. Based on the parameters that it learned in training, ChatGPT generates a response by predicting the most likely sequence of things that you might want it to do. An answer to a question, an image, so on and so forth.

15:52Each one of these steps is extremely demanding. Processing hundreds of billions of these parameters, learned patterns from ingesting training data, such as how the English language works or what a dog looks like, to produce even the simplest thing. Training these models is equally intensive, requiring ChatGPT to process massive amounts of data, another problem I'll get to in a bit, adjusting those hundreds of billions of parameters and developing new ones based on what the data says. As it quote-unquote learns more, though as we're clear, ChatGPT doesn't learn anything, it just makes new parameters to read things.

16:26A model like ChatGPT grows, making it more complex, which in turn requires more data to train on and more compute power to both ingest the data, create more parameters, and turn it into something resembling an answer. And because it doesn't know anything, it's suggesting the most likely to be correct answer, which leads it to hallucinating incorrect things that, based on probability, kind of seem like the right thing to say. These hallucinations are the dirty little secret of generative AI, and are impossible to avoid thanks to the fact that every single thing these models say is a mathematical equation rather than any kind of intellectual exercise.

17:03If you ask ChatGPT how many days there are in a week, it doesn't know that there are seven days, but it's been trained on patterns of language and generates a result based on those patterns, which at times can be correct and can also be wrong. There's no way of fixing this problem. You can mitigate it, you can make it less likely it will mess up, but hallucinations will happen, because there is no consciousness. It is not learning anything. This thing has no knowledge. more computing power would allow it more parameters to give it more rules so that a generative ai will be more likely to give a correct answer but there's no eliminating them and doing so may require more computing power than actually exists or is possible without an ai of consciousness an impossible dream known as average generalized intelligence that sam altman would have you believe is imminent there's really no solving hallucinations When you answer questions using probability, you're always going to have mistakes because you're not actually answering them using knowledge, intellect, or experience.

18:07You're using dice rolls. It's a bloody game of Dungeons and Dragons. We turned in Carter into Dungeons and Dragons. Anyway. A newly published paper by Teppo Fellin and Matthias Holweg of the University of Oxford agrees, finding that large language models like ChatGPT are incapable of generating new knowledge. It's a remarkably in-depth rundown of the fundamental differences between a large language model and a human brain And it combines both psychological and mathematical research going back to child psychology as well The basic building blocks of how we consume and learn things and how we make decisions as a result The paper Titled theory is all you need ai human cognition and decision making argues that AI's data and prediction-based orientation is an incomplete view of human cognition.

18:57And that the forward-thinking theorizing of the human mind, in layman's terms, the mess of the information we've learned over our lives, our experiences, and our ability to look forward and consider the future is just fundamentally different to a model that predicts things only based off of past data. Think of it like this. If you've read a book and you might think about writing a new book based on those ideas, you're not remembering every part of the book. You don't have a perfect memory. And you're also constantly thinking about things as your day goes on. The human brain is a goddamn mess. Generative AI is in some level stuck in amber.

19:35Though the billions of parameters might change, the data never does. The way it consumes the data may be, but the data doesn't change. In essence, generative AI is held back by the fact that it can't consider the future and is actually permanently mired in the data of the past. Their largest problem might be a far simpler one, a far sillier one, a kind of an ironic one. There might not be enough data for these bloody things to actually train on.

20:18Look ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. The secret to steady business growth? Fast funding with Amerifactors. If you're a business owner in need of capital for payroll, inventory, or expansion, you need Amerifactors. Get the help you need even with less than perfect credit, including bankruptcies. Amerifactors provides ready cash flow through accounts receivable management. Get tailored solutions. Call today for a free, no obligation, no impact to your credit score quote at 800-884-FUND.

20:54That's 800-884-3863. Or visit amerifactors.com. Hey, it's Ryan Reynolds here for Mint Mobile. Now, I was looking for fun ways to tell you that Mint's offer of unlimited premium wireless for$15 a month is back. So I thought it would be fun if we made$15 bills. But it turns out that's very illegal. So there goes my big idea for the commercial. Give it a try at mintmobile.com slash switch. A upfront payment of$45 for a three-month plan equivalent to$15 per month required. New customer offer for first three months only. Speed slow after 35 gigabytes if network's busy. Taxes and fees extra. See mintmobile.com.

21:32While the internet may at times feel limitless, a researcher recently told the Wall Street Journal that only a tenth of the most commonly used web dataset, the common crawl, a freely available 250 billion page dump of the web's information, is actually of high enough quality data for large language models like ChatGPT to actually train on. Putting aside the fact that I can't find a single definition of what high quality actually means, the researcher, Pablo Villobos, suggested that the next version of ChatGPT would require more than five times the amount of data it took to train its previous version, GPT-4.

22:09The new one It's called GPT-5, by the way. And other researchers have suggested that AI companies are going to run out of training data in the next two years. Now, that sounds dire, but don't worry. They've come up with a very funny and extremely stupid idea to fix it. One specifically posed by the Wall Street Journal is that the AR companies are going to create their own synthetic data to train their models, a computer science version of inbreeding that researcher Jason Sadowski calls Habsburg AI. This is, of course, an absolutely terrible idea A research paper from last year found that feeding model-generated data into models to train them Creates something called model collapse A degenerative learning process where models start forgetting improbable events over time As the model becomes poison with its own projection of reality The paper, called The Curse of Recursion, Training on Generated Data Makes Models Forget Highlights an example where feeding a generative AI its own data eventually destroys its ability to answer questions.

23:10And within nine generations, one answered a simple prompt about architecture with an insane screed about jackrabbits full of at symbols and weird characters. So not to worry again, the tech overlords have come up with a great idea to fix this problem. Their common retort to the problem of synthetic data is that you could use another generative AI to monitor the synthetic data being fed into a model to make sure it's right. At this point, I'd like to get slightly angry. Are you kidding me? Are you fucking kidding me? You're saying that the way to make sure the data generated by an unreliable generative AI is to use another generative AI, one with the same goddamn problems, which also hallucinates information, that knows nothing.

23:56You're going to use that AI to monitor whether the data that is created by an AI is any good. Are you completely insane? Are you insane? You're going to feed the crap from the crap machine into another crap machine to make it not make crap? Why am I reading journalists credulously printing this ridiculous solution in the New York goddamn Times? Every time, every time these bubbles are inflated because tech executives are able to get their half-arsed, half-baked solutions parroted by reporters who should know better. You don't have to give them the benefit of the goddamn fucking doubt. This is how we got the bloody matter of us.

24:30Pardon me. I've calmed down now. Anyway, anyway, if you're worried about model collapse, you're already too late as these models are likely already being fed their own data. You see, these models are trained on the web, as I previously told you, and they're desperate. They need data. They need more stuff. They need more stuff to ingest so they can spit out more stuff. The problem is that these machines are purpose-built to make a lot of content. And so the web's already being filled with generative AI. Generative AI is already spamming the internet. A report from 404 Media from last week said that Google Books has already started to index several different works that were potentially written by AI, featuring the hallmark generic writing tropes of these models.

25:19404 Media also reports that the same thing is happening over at Google Scholar, their index of scientific papers, with 115 different articles featuring the phrase, as of my last knowledge update, a specific phrase spat out by generative models. This is really bad, by the way, and this is only going to get worse. When you have an internet economy that is built so that the people that can put the most out there will probably get the most traffic, they're going to use these tools. These tools are great for that. If you don't give a rat fuck about the quality. This is the best thing in the world for you.

25:54And that's the thing. This is a problem both created and caused by these models. You see, the other dirty little secret of generative AI is that these models unashamedly plagiarize the entire web, leading outlets like the New York Times and authors like John Grisham to sue OpenAI for plagiarism. While OpenAI won't reveal exactly what their training data is, the New York Times was able to successfully make ChatGPT reproduce content from the newspaper, and the company has repeatedly said that it trains on publicly available data from the internet, which will naturally include things like Google Scholar and Google Books.