Unveiling Stanford's Clone: Exploring the $600 ChatGPT Reproduction

0:00With companies like Microsoft investing over 10 billion dollars into ChatGPT, it kind of begs the question, how valuable is one of these large language learning models? So today on the podcast we're going to talk about something very interesting that recently came out which is the fact that some researchers at Stanford said that they have essentially been able with some new technology that came out to clone ChatGPT for less than$600. Now when we're looking at Microsoft in 2017 investing a billion dollars getting this kind of technology kicked off and then just recently investing$10 billion into the company to further its efforts.

0:40It's pretty interesting that, you know, people are claiming they're able to duplicate this for$600. So on the podcast today, we're going to dive into what exactly happened and what this is going to mean for AI in the future. So first off, Stanford is calling it Alpaca AI. And essentially what they did was Facebook recently or Meta, they recently came out with what's called Lama model. So essentially, it's Facebook's like open source language model, It's pretty much the smallest and cheapest of all of the Lama models available. And it's pre-trained on a trillion quote-unquote tokens, which tokens is not necessarily words.

1:20It's kind of like bits of words or half. I don't know. It's just like how AI models measure what is being spit out and what it was trained on. So like one long word might be like three tokens and like a short word might be one token. Anyways, not that important. um so pretty much this model has a certain amount of capacity baked into it but it would lag significantly behind chat gpt with most tasks um and the the number one area is cost so the biggest competitive advantage in the gpt model comes largely just from its the enormous amount of time and manpower that open i has put into the post training so it's one of the things um and so like for example it's just like one thing to have read a billion books but it's another to have chewed through large quantities of questions and answers, conversion pairs that teach an AI what their actual job will be.

2:12It's not just the data that went in. It's about when you ask it a question and it gives you a response, kind of fine-tuning those responses after it has all that data. And that's kind of the value of OpenAI. So with the LAMA 7B model up and running the Stanford team pretty much asked chat GPT to take a hundred and seventy five human written instructions and output pairs and started generating more and and to start generating more like it in the same style and format and it was asking for 20 at a time so it was automated through one of open AI's really helpful APIs and in a pretty short amount of time the team at Stanford generated around 52 ,000 sample conversations to use in post-training the Lama model.

3:02So they generated pretty much all of this for around$500. So essentially what they did is they went to OpenAI and they asked it to come up with 52 ,000 sample conversations where it was asking questions and giving responses. So they used content specifically from OpenAI and then they fed this content into a Lama model. And the reason why that's important is it's because they didn't just feed it raw data from the internet. They didn't just feed it like, you know, billions of words from books like OpenAI originally was trained on. They trained it off of actual responses from OpenAI. So then they used all that data and they fed it into Facebook or Meta's new Lama 7b model.

3:46And they used that to kind of fine tune it. And the process for that took about three hours and it cost them less than$100. So all in their, you know, they're like 600 bucks into this, their$500 training or getting all the outputs from OpenAI and then about$100 training on Meta's new Llama model. So next they tested the resulting model which they generated, which they called Alpaca, against ChatGPT. And they had a bunch of different, I don't know, tests that they made for it. And some of them were email writing, social media, productivity tools. and alpaca their model 170 of the tests and chat gpt 189 of the tests now of course that's not very significant because essentially they're the exact same and statistically right like essentially all they're saying is what they were able to recreate was exactly as good as chat gpt and that doesn't make uh that's not very surprising since what they created was literally trained from outputs of chat GPT.

4:48So no shocker there. But what I would say what is pretty incredible is the fact that they were able to create something really powerful using chat GPT. And essentially, the implications of this are that anyone in the world can, you know, there's no reason why they have to stop at 600 bucks, like maybe you make it maybe spend$10 ,000, or maybe you spend$100 ,000 and make something a lot more powerful. But anyone can go and use outputs from chat GPT to train a new AI model on Meta's lama function, Meta's lama model. And essentially, you can create your own AI model. And the reason why this is important is because a lot of people right now are complaining about biases or the safety features built into OpenAI.

5:36And essentially, anyone that knows how to use these tools can now go and create AI models that are, you know, that are essentially unlocked, right? Like they don't have the built-in safety features. You can use them for whatever you want. You can ask any questions. A lot of people are raising concerns about, you know, maybe hackers can use this or people can use this for phishing or unethical things. You know, some people are happy about it because they think OpenAI went a little overboard on their trust and safety kind of things and say, you know, people are saying, look, I just want an AI that's going to tell me the truth based off of whatever it calculates, It's not based off of whatever its truth is, not based off of whatever developer at OpenAI deemed was the socially acceptable response.

6:19So, you know, there's all these kind of like there's all these kind of like censorships and different discussions going on in the moment. But the implications are very, very big. And essentially, they said that it behaved very similar to DaVinci 3, which is kind of came before ChatGPT and GPT 3.5 on a pretty diverse set of inputs. And of course, they also acknowledge that their evaluation is going to be limited on scale and diversity. It's not the biggest thing. But they said, you know, we really hope that they released the code and everything for people to clone and copy exactly what their process was.

6:55but you know they said there's no reason for people to stop at 52 000 questions you can go way bigger this was super cheap but the point is for 600 bucks anyone can make a fairly you know rough clone of chat gpt using some new technology that came out recently so it's gonna be really interesting to see what happens in the industry and i think this is also pretty interesting because another group recently said it managed to pretty much eliminate the cloud computing costs of a lot of these OpenAI models, and they released more code on GitHub that essentially they can run on a Raspberry Pi, which is just like a small chip.

7:32And they can complete the training process within five hours on a single high-end NVIDIA RTX 4090 graphics card. So you can train this without using complex, expensive cloud computing. You can train models. They can do some pretty impressive things, and they're essentially unlocked. Now, it is important to note that with all of this, you know, new technology and stuff going out, it technically is against OpenAI's terms of service and privacy policy to take, it's against their terms of service to take an output from OpenAI and to use it to train something that competes with them. That's in their terms of service.

8:16But, you know, I think it's going to be pretty up to interpretation. if people are able to you know so here's another interesting thought about that though is like they'll probably have different ways to try to say oh this was trained from us but it's like well opening i trained their data from everything on the internet whether or not those people always wanted it so you know who owns this content who owns a content uh that was trained from let's say reddit and then to an ai model then it gave an output that was trained another ai model and gave an output that trained another ai model like who owns the output that's gonna be pretty interesting to see how that all shakes out.

8:52That being said, though, number one, there's that aspect where OpenAI said, you know, you're not allowed to use outputs from us to train new AI models, but they compete with us, though. So, you know, if you're training one that doesn't compete with them for something, then maybe you're good. That'll be left up to decide. And also, it's going to be interesting to see how they try to actually detect that, right? Because we've had this cat-and-mouse game of people trying to do AI detection for a while now, and maybe people take all the outputs and they run them through something like Quillbot and they trained their AI model, and now it's essentially undetectable from open AI.

9:24So it's going to be really interesting to see how that shakes out. And then in addition, I would say the one other thing is that Facebook itself has limitations and rules around how its LAMA model is used. And they said that they're only letting academic and researchers use LAMA under non-commercial licenses at this stage. But that's also essentially moot since the entire llama model was leaked on 4chan about a week ago After it was announced so, you know Or I not a week ago, but I think it was a week after it was announced in any case It's gonna be interesting to see what happens. We have The ability to train models super cheap thanks to meta We have all of the data of a pre-trained of a model that's been trained well Thanks to open AI and then we have people that have figured out how to run this whole thing on raspberry pies instead of expensive cloud computing.

10:16So I think these three areas are going to come to a point and we are going to see some pretty insane, essentially, I think everyone's going to be getting AI models specifically trained on how they like, which is going to be interesting because it's, you know, some people are complaining that this is going to just perpetuate the echo chamber effect we see online and within social media where essentially everyone wants an AI model that's trained to their own beliefs and biases about the world. And I mean, it makes sense. People are going to want something that agrees with their ideological perspective on things and the way that the ai model is currently coming out it has ideological perspectives and so if those align with you fantastic if they don't align with you people are going to be disenfranchised and inevitably are going to try to get something that aligns with them more so it'll be really interesting to see what happens to see how people try to push back open ai and facebook and you know these other companies but i think this is going to be a really incredible space to watch in the next little bit and we are about to see an explosion in AI models that are coming down the pipe.