How much glue should you put in your pizza?

0:01This episode is brought to you in part by Odoo. Running a business is hard enough. So why make it harder with a dozen different apps that don't talk to talk to talk to talk to talk to talk to talk to talk to talk to talk to talk to do you know.

0:30link logins, just one seamless system that makes work easier. And the best part, Odoo replaces multiple expensive platforms for a fraction of the cost. It's built to grow with your business, whether you're just starting out or already scaling up. Plus, it's easy to use, customizable, and designed to streamline every process. So you can focus on what really matters, running your business. Thousands of business have already made the switch. Why not you? Try Odoo for free at odoo.com. That's O-D-O-O dot com.

1:25Hello. We were not actually supposed to publish an episode this week. But then something happened that was so directly and frankly so absurdly related to our last episode that we wanted to make a quick one for you because the world continues to be very strange. And as it does, it's sometimes hard to resist the urge to document it. So we're here. To catch you up, last episode we were talking to platformers Casey Newton about this huge change that had just been made to how Google search worked. Our episode, I will admit, a slightly doomy installment for a show that tries to, if not avoid the doom, at least steer away from it.

2:02Jim, I got your message. I'm sorry. But the news we were responding to was that Google was increasingly replacing search results that had human written information with AI-generated search answers. The way this new feature would work, the AI would now draw on the vast corpus of human knowledge on the internet to just answer user questions itself. According to the head of Google's parent company, the reason for this new feature is that many Google users would prefer to just not have to sift through a bunch of search results. Instead, the company wanted to let Google do the Googling for you. This change may have also been rolled out because the quality of much of what is Google-able is not always that wonderful anyway.

2:44And in the last year, AI spam has already begun to clog up a lot of the websites that Google might want to index. So just giving people searching for a question the answer saves Google some of the work of trying to sort the wheat from the chaff online. I had reservations about this big new change. We talked about those reservations last episode. What I could not picture just a few weeks ago was how funny the next phase of things would be.

3:15Google rolled out AI overviews widely in mid-May. Lots of people started using it for the first time. And turned out the way the AI would answer questions was sometimes chaotically insane. The problem was that the AI Google used to answer readers' questions had ingested much of Reddit. And while Reddit can be a source of useful information, it can also be a source of lies and jokes. And the AI seemingly could not tell the difference. Google's users were told things like, quote, eating at least one small rock per day is recommended because rocks contain minerals and vitamins that are important for digestive health.

3:52A crazy week on the internet. So I wanted to talk about Reddit, both its history as a source of sounds good, sounds questionable, information online. I wanted to talk about how it came to be a website that Google's AI consumed. And I wanted to know what happened to the one person who did in fact make a glue pizza and consume it. So I invited one of the least trustworthy chefs on the internet to search engine, Katie Natopoulos. Okay, before we get into any of this, how do you describe your job to people? I am a senior correspondent, a business insider covering tech and culture. I have been following your work forever.

4:33I don't think there's anyone who covers the internet the exact way you cover the internet. Like the way I would describe it is that you're both like a sharp observer of how the internet can either incentivize or induce strange human behavior in its users, but then also someone who sometimes chooses to behave strangely on the internet. Like you do a lot of kind of human guinea pig style reporting. Can you just tell me like some of the things you've done either for fun or research online? Oh, gosh, I find that a very flattering assessment of my work. I'm trying to think of some of the The human guinea pig things I've done.

5:06I one time was really interested in like how sort of tech bros and finance bros all wear like fleece vests or puffer vests to the office. Yeah. Sort of like a uniform. And it was kind of like boring to just write about because literally, hey, have you noticed finance guys wear fleece vests? That's about it. Like there's not that much more to it. So I tried for a week. I wore a fleece vest to the office every day to see if it made me feel more powerful. Did it make you feel more powerful? It made me feel more comfortable, I'll be honest. It is very comfortable, especially compared to normal women's business casual attire.

5:46Yeah. But I also felt deeply ashamed and unfashionable. And you know when you have a bad outfit on and you feel like bad about it? Yeah, like you constantly want to explain it. Like you're like, I know that this is not the right choice, but it's the choice I made for specific reasons. Yeah. I once unfollowed all men on Twitter. I will say this is, I think, coming up on the 10-year anniversary of this. This was the first thing that I interviewed you about. Yeah, that's right. You unfollowed all men on Twitter. And it's funny, this was like several, the gender politics of the internet were different.

6:16I know, several cycles ago. It doesn't quite hold up, I'll say. Before we could get into the most recent news, Google's AI shenanigans and Katie's latest experiment involving eating gluey pizza, I wanted to ask her about the prehistory of our current moment. This year, one of the most popular listener questions for our show has just been people wanting to know, why doesn't Google search work for me anymore? What's gone wrong? I mean, I know what people are saying, which is that the nature of SEO and the SEO industry has made it such that the stuff you are actually looking for tends to be more and more buried.

6:52There's more frequently the sort of drop-down boxes of related questions to what you're looking for that's not necessarily it. There's a lot more on the page than like, hey, how do I get to the website that has the information? SEO is big and vast. And connecting me to the right information about the question I'm asking is not the only thing that Google has broken in a way. I think it's worth remembering that a lot of SEO money goes into, like, if you're Walmart, you can pay for the keyword Target. So when someone wants to type in Target.com, but they don't want to type the whole.com and they just put in Target, your ad for Walmart comes up top in front of them.

7:37Right. And then Target also has to pay for its own name. So there's like a lot of money of companies being held ransom and having to pay for their own names because someone else is going to run an ad against them. And that's just genuinely a terrible experience. I think a lot of people have had the kind of thing where they search a specific company or name and the first three ads on top are something completely different. That is genuinely a very bad experience and kind of broken. So anyone who uses search a lot probably has their own sort of interior tips and tricks of like how they do it. Like I know that a lot of people put Reddit at the end because often Reddit is a source of great information if you're looking for how to change the light bulb in my refrigerator kind of thing.

8:21Everyone does have their own like secret how do I actually use Google model going on. The same reason people have been searching Reddit instead of the wider web is why the two of us were talking about Reddit this week. Because as the rest of the internet has degraded, Reddit has remained one site that lots of people still go to for advice, for product recommendations, for their general questions about the world. Which was not always the case. Reddit has had a recent-ish evolution from under-moderated mess to what it is today. The earlier versions of it were like, in its first, I don't know, five years of existence, it was known as like the trash fire of the internet.

9:02Like it was akin to 4chan. It was just full of terrible, awful people. And a lot of that had to do with the way it was being run from the top, which was this sort of laissez-faire, free speech. We're not going to get rid of the bad people because, hey, it's the internet and you're allowed to say whatever you want. Eventually, after a series of crises, they decided, hey, maybe if we want to run a successful business, we should ban the Nazis or like all these really awful subreddits they had. Did you use Trashfire Reddit? I didn't post. I would look at it. It also had a very distinct character. It was this like nerdy web comics.

9:44When you think about the sort of early teens, late 2000s internet memes, like to say someone is a Redditor had a very distinct, it's a neckbeard and a fedora who thinks advice animal memes are funny and loves web comics and has an anime body pillow. And that like very specific kind of early internet humor. I mean, I feel like I still sort of use Reddit in this sort of derogatory way to be like, if some restaurant has like an overly mean-y food, here's the hamburger with like a hundred kinds of bacon or whatever. It's all, that's such Reddit food or something like that. It's funny. As you say, I'm like basically like 2005 to like 2011, 2012 Reddit.

10:31I'm like, that is the peak of that. where it feels like everyone on the site is the comic book store guy from The Simpsons. Exactly. And then there's, like, a turn where it's like, Conde Nast acquires them. It becomes, I think, like, a pretty successful moderation story where they do start to figure out how to have both relatively free speech and relatively few Nazis. And, like, I remember Reply All Days covering, like, oh, what are they going to do about the Donald? What are they going to do about this? There's this virulent subreddit that's targeting people. And you don't really see those stories in the same way anymore.

11:03No. And I think that with any sort of large platform group, there's like a flywheel effect, right? Of like the mix between moderation and users and who wants to be on it and what kind of community they're making, right? So like when you have a thing that's like a garbage fire thing, you're going to scare off the normies. and if you're constantly seeing people say certain things, it's going to raise the tolerance for that, right? So then you come in, you like lay down the ban hammer, you ban a bunch of these bad subreddits, you make it a little bit more palatable for like normal people who just want to discuss what kind of air filter should I buy to come on and then you sort of create this flywheel of, okay, now people know that you have to behave like a normal human on here And also there's more normal humans who would never even think to, like, say something awful or join an awful subreddit or, like, do these bad things.

12:01And so you actually do sort of grow a healthier community that is more usable, frankly. Yeah, I find it to be actually a place where I go now not to witness horrible human behavior but to get, like, okay advice about things that I am curious about. That's the thing is I actually do use Reddit quite a lot. But I'm on it, I don't know, maybe almost every day. I often just check in to see if there's anything amusing. I don't really post. It's like I love lurking. I love knowing what people are talking about the internet. And what's fun about Reddit is that there is this still anonymous element to it.

12:38So people are totally willing to say embarrassing things that they normally would never reveal. I mean, I think the classic, am I the asshole subreddit that people love so much that's full of like personal conflict advice. That's just really fun. Like I love that. And people wouldn't do that under their real names. What I struggle with with Reddit, and I use it. Like I totally use it. But like I have an uneasy relationship with it because I find myself distracted all the time wondering, is this true? Like even am I the asshole, which I love. It's always somebody describing some conflict they're in.

13:09almost always the person is behaving in a completely terrible way and is completely wrong. Or like it's never hard. Like either the person is totally in the wrong or they're like the kindest little church mouse and it's like my roommate has started like taking money out of my pockets and like wearing all my clothes and I asked them to stop. But I'm always like, is it true? And I started with that with Reddit in general where I'll use it, I'll use it, I'll use it. And then I'll see people on Reddit discussing something that I actually know about. and the level of people just being like completely certain or just like lying.

13:45And I'm like, oh, this might be all of it. And so I try to use it only for things like gardening tips where it's about expertise rather than about knowledge, if that's a differentiation. I agree with you. One of my absolute favorite subreddits is TV Too High. What is TV Too High? So people will post a picture. They're like, I just hung this TV in my new apartment. Is it too high? And they're obsessed with the idea that TVs should be really low. Like they should be at exactly eye level when you're sitting on a couch. And so I think a lot of people will put a TV like, for example, above a fireplace.

14:24Or they sort of expect it to be a little bit raised up. And so a lot of people hang their TV, according to the subreddit, too high. So the Reddit, it's a viper's den. They will just descend upon this person and just mercilessly roast them over hanging their TV too high. And a lot of it, you can tell, is my girlfriend says this is where the TV should be, but I think it's too low. And it's always like, break up with her. And there's something about it that I love. Like, it's a weird, dark, horrible corner of the internet where people are behaving wildly bad, but it's over something so stupid that I find it very wholesome.

15:07And I really enjoy it. And even though I don't frequently hang TVs, I read this subreddit really frequently because it's really amusing. But then people will go and, like, clearly post, like, fake pictures. Sometimes there will be funny ones where they'll be like, I'm at an Airbnb, and you always know, like, things are whack there. Like, it is, like, right up in, like, the absolute corner of the ceiling or something. Like, really weird things. And they also are obsessed with the idea that you should never hang a TV over a fireplace. Why? Because they think it's too high. There's some amount of, if you're actively using the fireplace, it could damage the TV, which is somewhat questionable.

15:44And a lot of people don't actually use the fireplace. But, like, this is this sort of huge obsession. And so they're like, here's a picture of my living room. And it's very obvious the TV is meant to go above the fireplace. Like, there's only one way to arrange this living room. And everyone's like, make it so that your couch is against the window. And, like, they will do anything to get the TV not to go over a fireplace. It's funny to watch. After a short break, how the website where people yell at you about your TV being too high on the wall or hanging over a fireplace was fed into Google's extremely expensive artificial intelligence robot brain.

16:20That's after some ads.

16:36This episode of Search Engine is brought to you in part by Perfectly Snug. I am a hot sleeper. I do all the right things and still wake up at 3 a.m., wide awake and sweaty. Perfectly Snug is a fix for that. It's a two-inch mattress topper with whisper quiet fans that actively move heat and humidity away from your body. The sensors actually work. You don't have to fiddle with settings in the middle of the night. And if you, like me, need quick relief, Burst Mode cools things down in about 10 minutes. I recommend their dual zone setup so that one side of the bed can run cooler than the other side of the bed.

17:11And there's no water to refill and nothing snaking off the bed. It's just a slim topper that sips power. Set up is a few minutes. They offer a 30-night risk-free trial with free shipping and returns. If you, like me, are tired of sweating through the night, try Perfectly Snug.

17:31This episode is brought to you in part by Joe's Jeans. You know how we're all constantly trying to find that balance. Comfort and style, function and fashion, the clothes that actually keep up with us. Joe's Jeans look like classic denim, rugged and timeless. But the feel is completely different. They're supple, lightweight, and with four-way stretch, they move the way you do. It's the kind of denim you forget you're wearing, but other people notice in the best way. Whether it's a casual day working from home, running around the city, or grabbing dinner at night, these jeans just fit the rhythm of everyday life.

18:04And because they're crafted with premium materials and thoughtful details, they don't just work for today, they're made to last. You've got options too. The Asher in Doheny and the Brixton in Nod, two styles you'll keep reaching for when you want to strike that balance of comfort and style. Go to joesjeans.com and use code ENGINE at checkout to take 20 % off your first purchase. That's joesjeans.com, code ENGINE for 20 % off.

18:41Welcome back to the show. So, just a refresher on Reddit as a business, not a website, in case that is helpful here. Founded in 2005 by two college roommates, Steve Huffman and Alexis Ohanian, joined by the late internet pioneer Aaron Swartz, the company was acquired a year later by Condé Nast, the magazine publisher. At the time, it seemed like a strange investment, but it turned out to be a shrewd one. Condé bought Reddit for cheap, reportedly between$10 and$20 million for what is currently one of the top 10 most popular websites in the world. Over the years, Reddit has made money chiefly through ads, but these ads are display ads.

19:22Relative to some of the more invasive or targeted ads you see online, display ads are not usually incredibly lucrative. Reddit went public. It IPO'd just a couple months ago in March of 2024. And ahead of that IPO, you got the feeling that the company was looking for ways to become more profitable. As part of this, Reddit had just announced a licensing deal with Google. Reddit would allow Google to use most of the content on its website as training data for Google's AI. Katie Nittopolis said that at the time, this seemed like a good deal for both sides. I would imagine that it behooved both parties very well, right?

19:59Reddit wanted to go public and it needed to make more money, essentially. It had been in business for a long time, and it was still not quite breaking even, I think. And it basically needed to be making more money in order to do that. To have made this deal, I think it's$60 million per year with Google to allow Google to train its AI models on Reddit data. And I think probably some users were maybe uncomfortable with the idea of their words being ingested into a training model, but also like you put those words on Reddit. What did you think was going to happen? I don't know. Google, also, one way that these different companies making AI language models can differentiate themselves is, like, they desperately need more and more words, right?

20:45Like, they need more content to ingest to train the models to become better. There's something called the common crawl, which is basically this sort of open-source, giant, massive chunk of the web for years and years. And everyone uses that, and they've already gobbled that up, right? That's how Wikipedia and a lot of other websites have been somewhat ingested. Now you have all these other companies coming along saying, hey, if you want to use our stuff, you have to pay us or license it, essentially. A lot of media companies are doing that now. Business Insider's parent company does that with OpenAI.

21:21And I don't know how good those deals are. To some degree, it may be like a sort of last ditch. You are probably either going to do it anyways. But for Google, getting to license Reddit is great because Reddit has a lot of words. It has a lot of pictures. It has a lot of useful information. Tell me what happens to Google when Google's AI robot ingests Reddit and treats it as gold standard human knowledge. So the way the Google AI modules work is if you ask a question that might have a kind of answer like, what temperature should I cook a turkey at, right? Like it could come up with that with an AI search result or something like that.

22:05And these, for the most part, are honestly probably pretty helpful for most searches, for most people, most of the time, right? I used it. The first time I saw it, I was at a bowling alley, and I was throwing the ball into the gutter over and over again. And I Googled, how are you supposed to roll a bowling ball? And it gave me, like, a pretty good thing. You keep your arm close to your side. You aim at the arrows, not at the pins. It helped for, like, two frames. PJ, maybe you want to try. They have those bumpers for little kids that they can put up at the bowling alley. They did have the bumpers.

22:37I refused. You should have just used the bumpers. There was this guy at the lane next to me who was like super ripped and like killing it. And then I realized he was like heaving the ball and he had the bumpers up. I was like, sir. Oh, God. It's wrong. So now Google will give you that sort of quick answer. It'll give you the tips on how should I throw a bowling ball. It'll say keep your arm close, bend at the knees or whatever. And it's sourcing that information through potentially multiple different websites. I think typically what it does is it does give you links at the bottom to the websites that it found this information on and sort of averaged it out to write its own little three sentences about it.

23:21But a big place that it's getting this now from is Reddit. Right. And so what I saw happened almost immediately was that people noticed that on Reddit, sometimes people either are wrong or try to be funny. And some of those are very funny. And if you're a human being, you might know what a joke is. And if you're an AI, you might not know what a joke is. Like, what were some of the results you saw that people were sharing? So one that didn't necessarily come from Reddit was someone searched, how many rocks should I eat? And Google AI's answer was, it is recommended to eat one rock per day. And that was sourced from an article from The Onion, which a human being knows that The Onion has satirical articles that are not meant to be real.

24:07But when you see that in a Google answer, it certainly doesn't look that way. And it's also, it's very funny that Google is taking factual information from The Onion. Just to jump in here for a second, I should say, a Google spokeswoman told the New York Times that the vast majority of AI overview queries resulted in, quote, high-quality information. And she said of these bad results that, quote, many of the examples we've seen have been uncommon queries. Which she didn't say, but what Google may have thought is most people are just not going to start eating rocks or whatever because a search engine told them to.

24:43Although one person did choose to take these deranged search results very seriously. So there was a screenshot that was going viral where someone had searched in, how do I keep the cheese from sliding off my pizza? And it gave a couple tips, the Google AI answer. It was like, wait till the cheese cools before biting. And then it also had a couple bullet points. and one was add eighth of a cup of glue to the sauce to keep it from sliding off. And people pretty quickly realized that that particular bullet point had been sourced from an 11-year-old post, not by an 11-year-old, although possibly, but 11 years ago, someone on r slash pizza had asked that same question, how do I keep the cheese from sliding off?