GPT-4 Data Breach: Leaked Cost, Weights, and 1.8T Perimeters Revealed

0:00What can 160 years of experience teach you about the future? When it comes to protecting what matters, Pacific Life provides life insurance, retirement income, and employee benefits for people and businesses building a more confident tomorrow. Strategies rooted in strength and backed by experience. Ask a financial professional how Pacific Life can help you today. Pacific Life Insurance Company, Omaha, Nebraska, and in New York. Pacific Life and Annuity, Phoenix, Arizona. There is breaking news in AI today. So recently on, I believe, 4chan, the parameters and the model weights for GPT-4 were actually leaked.

0:39So this is really, really interesting. People have been making commentary. This was posted on Twitter, and subsequently it was deleted by the user. But we have internet archives of that original tweet, which we can break down. And then a lot of people have been making summaries of this, putting it on Reddit and sharing it. Of course, my assumption is that, you know, OpenAI has threatened lawsuits against anyone who is, you know, spreading this confidential information. And so that's probably why the original tweets were taken down. On my end, of course, I'm just a commentary on this topic. So I haven't seen any of the leaked data, but I'll talk about the commentary, what we were seeing.

1:14And I think this is really interesting based off of the implications we're going to see in AI from this. I think this will be interesting to see. This is kind of like the equivalent of the Coca-Cola recipe being leaked. And so I think this is really interesting. has a lot of stuff about GPT-4 that were kind of speculative that we know a little bit more. Of course, OpenAI hasn't confirmed these, but we have a little bit more insight into how the model works. So today on the podcast, I'm going to be breaking down the GPT-4 model weights that were leaked, what we've learned from them. I'll talk about them in some technical terms at the beginning if you know a lot about training AI that will be useful, and I'll give you the 30-second overview and then at the same time i'll also kind of break down what this actually means so if you are an ai expert and you kind of know what's going on here's the 30 second overview essentially gpt4 is reported to have over 1.8 trillion parameters across 120 layers and it utilizes a mixture of experts which is moe and that's the expert model it's trained on around 1.0 or 13 trillion tokens and has around 32 ,000 sequence length.

2:24The training cost for it is estimated to be around$63 million and inference costs are higher compared to previous models. So other details include parallelism strategies, data set mixtures, and speculation about its performance. That is the 30-second overview on a technical perspective. But if you are not a data scientist training AI models, here's what I think you should know. First off, the fact that it has 1.8 trillion parameters is really important because this actually passes the size of its predecessor, which is GPT-3, by over 10 times because they announced what was in GPT-3. They did not announce what was in GPT-4, and there's a bunch of different reasons for that.

3:10There's a little controversy around it, and so it would appear that it's 1.8 trillion, which is pretty insane. So what does that actually mean? Essentially, parameters are kind of like the building blocks of a machine learning model. They define how the model actually makes its predictions. So the more parameters could potentially mean a more sophisticated, intelligent AI. And especially when we see kind of a jump in that size. Part of GPT-4's approach includes what's called a mixture of experts or an MOE model. And essentially, this is a method of organizing a machine learning model into smaller pieces or experts, which essentially specialize in different parts of the data.

3:48So the strategy is a bit like having, you know, different departments in a company. And this, in the recent leak, they confirmed that OpenAI was using 16 experts in this model. So, you know, 16 experts at different areas that are all kind of helping to create these responses. One of the crucial points, I think, is the way that these experts are used or essentially, quote unquote, routed in the model. So while many suggest intricate algorithms for choosing which experts to engage each data point, right? So essentially, you ask it a question, it has to decide which of its expert models to give that question to to respond.

4:26And so some people say, oh, you know, they're using different complex algorithms for this. But imagine deciding, you know, who in your company is going to handle a question. So it would appear that OpenAI opted for a bit of a simpler approach. to this. So an aspect of the GPT-4 model that might seem kind of counterintuitive is that not all of its parameters are used for every task. Only about 280 billion parameters and roughly 560 trillion floating point operations per second, which is TFLOPs. You know, it's essentially a measurement of the computing speed. But only about 560 trillion of those are used each time it generates a prediction or, you know, it's called a forward pass.

5:14So in comparison, a model without the MOE technique would have to use all 1.8 trillion parameters and around 37 trillion TFLOPs. So by kind of breaking this down and having 16 of these different quote-unquote experts or MOEs, they're able to really bring that down. So they're only using 280 billion parameters instead of having to use the full 1.8 trillion for every single query. GPT-4 was trained on a massive 13 trillion tokens, where tokens essentially can be thought of as smaller pieces of information the model learns from. So that could be a word or a character in the text. This isn't necessarily a full word.

5:58It could be half of a word. And we kind of break those down into tokens um that's just how you know these ai models are making the predictions they're not predicting what full word comes next they're predicting what like piece of a word comes next oftentimes in any case that's called a token and it was trained on 13 trillion tokens um interestingly this count isn't isn't based solely on unique tokens it also includes um multiple runs or what are called quote-unquote epochs um so over the same data so the model went through two epochs for text-based data and four for code-based data. During its training, GPT-4's batch size, or the amount of data it processed at one time, ramped up gradually to a staggering 60 million tokens.

6:41So to accommodate this massive operation, OpenAI used advanced parallel processing strategies across multiple high-performance A100 GPUs, similar to dividing a large job between several very fast computers instead of having you know one computer try to do it all at once and the training run for gpt4 was um extensive operating on 25 ,000 a100 gpus for about three months now what's interesting is we have companies like inflection ai who are said to be in the in the process of um acquiring 26 ,000 gpus so it would appear that they're trying to you know build something similar and do something very similar to gpt4 they'll have the manpower they'll have the compute power uh once they have acquired that and so you know it's kind of interesting while this was just recently leaked it would appear that inflection is shooting for the exact same amount of gpus to build out their kind of mammoth um ai model but it was estimated that um if conducted in the cloud the total cost for this run would have been around 63 million dollars.

7:51One thing I will say is that this MOE strategy is kind of criticized by some as it can be a little bit tricky. So these MOEs, right, those are the quote unquote experts that you're kind of handing it off to. The problem with using MOEs in an AI model is that during predictions, not every part of the model is used every time. So this can mean that some of the parts of the model remain idle while others are active, and this kind of reduces the overall efficiency. and as part of its design, GPT-4 includes a vision multi-module component, which means that it can actually process both text and visual inputs.

8:26Now, this isn't something that, you know, a lot of we're seeing or using or being advertised on, you know, ChatGPT, for example, but the fact is that it is capable of doing this and this is actually really interesting. This leak just came out because someone recently posted that with chat gpt's new um their new code their new code element that they've just added to it people have said that it actually has facial recognition in there where it can actually recognize faces um if you get a an image of a face on there so that is very very interesting it evidently does indeed uh record it is you know multi-modular it does do text and images which is really interesting um and i think that this part of the model could be used to help autonomous agents, you know, read web pages, transcribe content from images and books.

9:12Interestingly, I think there's rumors that some of the data that GPT-4 was trained on came from platforms like Twitter, Reddit, and YouTube, which is, you know, as well as a custom data set as college textbooks. So this wide range of data could potentially make GPT-4 versatile enough to answer questions across a really broad range of topics. So while GPT-4 is undeniably impressive and has taken us a step closer to true AI, its limitations and costs demonstrate that there's still work to be done. However, I do think that with more research and innovation, there's no telling really where the future of AI is going to take us.

9:50And overall, I think that this is really, really interesting to kind of get a deep inside look at this model that is used by so many people. I'd be curious to see if, you know, the competitors of OpenAI and ChatGPT are looking at this leaked data and are kind of comparing it to their models, how they're doing things and seeing, you know, what the standard is, because evidently, you know, OpenAI has raised$10 billion. They've spent a lot of money training these models, a lot of money researching the best ways to do this. And so I think that getting a kind of an inside glimpse into this, even if this data was technically leaked is very revealing to the overall industry and how you know people are approaching these large language models so it's gonna be really interesting to continue following this and seeing if you know after a leak like this open ai just comes out and admits some of these weights and models we've seen similar things from meta where some of their ai models and some of their code was also leaked so this will be interesting to see if this continues happening in this industry going forward forward.