The Data Dilemma: Sourcing Insights for ChatGPT

0:00So ChatGPT, as we know, relies heavily on data to generate responses to user inquiries. So where exactly does all of this data come from? In the podcast today, we're going to explore the various industries, websites, and datasets that provide ChatGPT with all of the information it needs to perform its task of generating responses. So one of the partnerships that it has is with Microsoft, which has provided a large dataset called Common Crawl. So this dataset is a massive collection of web pages, which includes everything from news articles to blogs to e-commerce websites. By training on this data set, ChatGPT is able to understand the nuances of natural language and generate responses that are more accurate and relevant.

0:49And it's important to note that OpenAI, which is a creator of ChatGPT, has been in partnership with Microsoft for a long time. About five years ago, I believe, it took over a$1 billion investment from Microsoft to help build out its new tool, which eventually became ChatGPT. And since ChatGPT launched, they've taken another$10 billion from Microsoft. So obviously very, very deep connection. So it's no surprise that Microsoft was able to help them get this common crawl, which I believe is a collection essentially of all websites on the internet that are indexed to be crawled by search engines. So I think the way that this works, and probably the reason why Microsoft was such a good partner for ChatGPT from a data standpoint is because Microsoft owns the Bing search engine.

1:36Just like Google, the Bing search engine crawls the entire internet and it crawls for websites. Now, websites can state, you know, they can put something on there called a no-crawl text for robots, which essentially says that these web crawlers like Bing and Google can't crawl them, but that's pretty much suicide for most businesses, right? if they're putting a no crawl thing on their website so they're not going to be indexed on search engines and they're not going to be included in google results and bing results but mostly probably they're worried about google results when they do this so um by partnering with microsoft openai was able to have a massive advantage in a sense that they're able to get all of this data right because when bing goes out and crawls the entire internet they're storing tons of this data essentially all of the content from these web pages on their servers and Microsoft is able to do this pretty well because they own a Microsoft Azure which is essentially like AWS or Google Cloud it's Microsoft's cloud division and it's a very fast growing division very lucrative division it does all of this computational power for a lot of websites a lot of website hosting and so Microsoft has this really powerful engine and they're able to harness it.

2:52It helps with their Bing search engine, obviously. And now that OpenAI partners with them, they were able to get that, which helped them to train ChatGPT. So in addition to Common Crawl, ChatGPT also uses a number of other data sets to improve its performance. And these include number one, Wikipedia. It's of course, probably one of the most well known encyclopedias. Wikipedia gives a ton of information on a wide range of topics. So by using Wikipedia as a source of data, ChatGPT is able to understand the context and background of many different subjects, right? So like if you go to a website, it might be talking about something, but when you want like the definition of a topic or an item or a person, Wikipedia is a really good place to get that, to get context on it.

3:38So I think that was a really big data set for ChatGPT. In addition, open subtitles. So this data set contains subtitles from movies and TV shows, which Chad GPT used to help learn about different dialects, accents, and speech patterns, which is very crazy, right? And, you know, when you go and you tell Chad GPT to, you know, write me an email in the tone of Woody Allen or in the tone of Buzz Lightyear from Toy Story, it's actually able to do that. And I think a large part of that is due to the fact that it was trained on open subtitles so it knows who all of these different actors and people are that are in movies and it knows how they talk, what their speech patterns are.

4:21Now it might not necessarily know what the audio sounds like, right? So you use other AI tools for that, but for their speech patterns and for how they talk and the words they use, that's where I was able to get it from. Another data set that Chai Chupu uses is the Gutenberg project. So this data set contains a large collection of books that are in the public domain. And this gave ChatGPT access essentially to a wide range of literary works that it uses to improve its understanding of language and writing styles, similar to the movies, right? Now you can get it to say things in the style of a specific author.

4:56But also, this really can help it train on different styles of literacy. And when you have just that many books, it's really, really powerful. So the Gutenberg Project was very powerful and very much like it is Google Books Ngram Viewer. So this data set contains a really large collection of books that have been scanned and digitized by Google. This is a massive project Google has been doing for a number of years now. And essentially by analyzing the frequency of words and phrases in these books over time, ChatGPT can gain insights into how language has evolved and changed over the centuries, which is a really powerful tool as ChatGPT looks at the evolution of language and how it works.

5:35I think it's kind of important to look at these data sets and what chat GPT gets out of them, because it really can help you understand just how vast, like when someone says, you know, chat GPT was trained on the entire internet, it is really hard to wrap your brain around what that means, what the implications of that is. And so these, some of these data sets kind of give us insight into how it works and how it was trained and, and the exact implications of each of these data sets. So another one, a very big data set for ChatGPT was Reddit. So while this is not a traditional data set, Reddit is a really popular social media platform that provides a massive wealth of user-generated content.

6:13So by analyzing the posts and comments on Reddit, ChatGPT gained some massive insights into the language and behavior of different online communities. And the reason Reddit is important, right? ChatGPT couldn't go and crawl Facebook because Facebook is a closed off platform. You need an account you have to be friends with someone to see it but reddit on the other hand is an open public forum so if someone starts a conversation on reddit about something all of the questions answers and comments are all public and the thing that is really powerful about reddit when it came to training chat gpt is that on reddit every comment can be upvoted or downvoted so chat gpt very easily can see you know the topic of conversation everyone's responses what responses are positive, what responses are deeply popular and deeply unpopular.

6:59And it really can get this like instant grasp on not just the comments that someone might have or responses to a topic, but everyone else's response to that how popular how many upvotes a comment gets how many downvotes comment gets, this gives it some really big insights. So beyond all of these data sets, chat GPT also relies on a whole number of industries and websites to provide it with up-to-date information on a wide range of topics. So these include news websites. ChatGPT can use news websites to stay up-to-date on current events and trends. Now we do know that ChatGPT has a specific cutoff date where it doesn't technically access the internet and use anything in the internet that was after the year, I think 2021.

7:42But that being said, it still has a lot of this data from the past and the future models are being trained on the most up-to-date information. another area is e-commerce websites chat gpt uses the e-commerce website data that it's gained to learn about different products and services as well as to understand consumer behavior and preferences right because when it looks at a website for example that sells a product it's able to see reviews it's able to see how many people have purchased that and so it's able to use that data from a website like amazon for example to see how many positive comments a specific product has and now it knows literally what the most popular products are it knows what products people are buying and knows what products people are reviewing, which products people are satisfied with.

8:24This is a really powerful insight it has into the world of e-commerce. In addition to Reddit, there's a lot of other social media platforms that it's able to scrape and look at. And in addition to that, ChatGPT also looks at academic research. So it looks at a lot of academic research to gain insights into different fields of study. And by analyzing these research papers and articles, it can gain a deeper understanding of various topics. Finally, government websites. ChatGPT uses government websites to learn about different laws, regulations, and policies. And this can be particularly useful in generating responses related to legal or political topics, which, while a lot of people have, you know, complained that ChatGPT and essentially the moderators of ChatGPT have kind of reined in anything it can say about politics, it does have the capability.

9:15And so when it tells you like, I'm just an AI model, it has that canned response, like I cannot respond to this topic. It's not because it can't because they've trained it to be able to, it's because the moderators have turned that feature off, which is controversial. And some people have positive and negative opinions about that. Overall, the data sources that ChatGPT relies on are super vast and varied. But by leveraging these different data sets and industries, it's able to generate responses that are very accurate, relevant, and pretty timely. So as natural language processing continues to evolve and improve, especially as we're looking forward to things like GPT-4 that is going to be coming out on the Bing search engine soon, we can expect to see even more sophisticated language models like ChatGPT emerge, and each will be relying on their own unique set of data sources and industries.

10:02My prediction is that after ChatGPT has launched and people are seeing how powerful and valuable it is, all of these new AI engines are raising millions and millions of dollars to compete. You know, Anthropic was funded by Google. They gave them$300 million after they'd already raised over$200 million. And those companies are trying to build these competitors. but my prediction is that a lot of these companies like reddit and these other companies that allow chat gpt to crawl for free are going to be closing down access to these large language models because they see how valuable their data is they're going to be charging for it in the future so i actually think that builds a big moat around chat gpt and open ai it actually makes it harder to crack there's a bunch of other big companies like you can look at google for example they're going to have that same crawl data that Bing was able to give ChatGPT, that same common crawl data set.

10:59And so big companies are going to be ahead at this point. So it'll be interesting to see how smaller companies play, how competitive the field gets. And it's going to be interesting to see what kind of monopolies of AI are coming out of this and if that is able to withhold or if smaller players are able to move into the market. One other prediction I have, hot take, is that even when people are going to be trying to lock, they're going to try to lock down their content and their data. I believe like you look like, look, every company in the world has been hacked. And if this content is super valuable, right?

11:32Like if people are paying hundreds of millions of dollars for inevitably, every data set in my mind will be hacked and consolidated will be sold on the black market. People will probably have different methods for washing this data, rinsing it, making it look like, you know, it isn't, you know, they didn't just steal it from places. But I believe that whether people tried to lock down their content or not, eventually, the data will get out. And we're going to see a vast amount of different language learning models that are able to produce tons of incredible content, whether that's for good or for bad.

12:05I know it's controversial, people have a different opinion. So that is my prediction on what is going to happen in the space, though. So it's gonna be exciting to watch and see how these large language models progress, how they get better, and it's really impressive to see the data sets that they're already working with.