Unveiling the Toll: Exploring the Hidden Costs of ChatGPT's Impact on AI Innovation

0:00Everyone knows that the cost of running AI models like ChatGPT are pretty high. A lot of people have complained when saying ChatGPT is going to beat Google or replace search engines because the cost of a ChatGPT question is about 10 times to 100 times higher, depending on what it is, than a simple Google search. So obviously, these are some pretty massive upscaling in the cost. So today on the podcast, we were talking about the costs of the AI, the AIs that we use, and essentially what people are calling the AI problem, the AI cost problem. So the main thing is that AI chatbots are losing money every single time you use them.

0:41And while some companies like Google and Microsoft can foot the bill right now, this is going to be pretty detrimental for the entire industry as a whole and innovation within the industry if no other competitors can get into the space. So today we're going to talk about what's happening in the space and why this is the case and what the hope is for some of these smaller breakout AI models that we're hoping to see. So I think one of the most important things to talk about is just that the expense and the limited availability of just the computer chips, right? We talk about NVIDIA a lot on the podcast and how much money they're making selling these GPUs and computer chips.

1:14And that is one of the biggest cost constraints for these companies being able to afford and run some of these massive AI chatbots that, you know, some would say, well, chat GPT has got to be making a ton of money. They got this$20 a month premium subscription. but one thing that is really interesting that not a lot of people are talking about is the fact that models that are being deployed right now these ai models as as impressive as they might seem right chai chupiti can get a lot done they are not actually the best models that's what tom goldstein a computer scientist professor at university of maryland has been saying i mean he says that as a result the models you have have a lot of weaknesses so he pits the issues um you know the bias or the blatant falsehoods, the things that we call hallucinations from AI models.

1:58He puts that squarely on the shoulders of the fact that it is so expensive to run these models. A lot of these companies are running cheaper, less effective, less powerful, less objective or true models because the cost of running the better versions are so much higher. So I think one example of this is Google, who when they first announced Bard, they said that it was going to be running on a light version. And the reason was because the full-on version, the capabilities Google has were just so expensive, even they didn't want to foot that bill, as some speculate. I think that for a lot of these big, huge tech companies that are really basing their future on AI, they rarely discuss the cost.

2:42And, you know, Microsoft, Google, they're all declining to comment on this issue. And a lot of people say that because everyone's talking about the ways that this AI is going to integrate into every vertical in every industry, they don't want people to slow down this integration because they're worried the cost will eventually go up, right? We look at a lot of companies that make the cost of their product quite cheap to get mass adoption, and then they slowly raise the price. Google and Microsoft don't want people getting the idea that that's going to be the case with their models. They really just want everyone to integrate as fast as possible.

3:15So they're kind of staying out of this entire discussion, but it is an important discussion to have. I think just the really intense computational power that these AIs use really made it so that when ChatGPT, you know, came out with GPT-4, that was not something they could give away for free. That had to be behind their paid version. Even to this day, if you want GPT-4, you're going to have to pay for it. You can run, you know, the weaker GPT-3.5 on ChatGPT for free. But there's a lot of limitations. As you know, September 2021 is a day of many curse because that's the last day that chat GPT was trained on.

3:51It doesn't know anything beyond that. But in addition to that, like I have experimented extensively with GPT four and the power is much, much greater than GPT 3.5. I always get like, I feel like the responses are like 30, 40, 50, 60 % better out of GPT-4. And we've seen that in a very literal sense, in a very measurable sense with GPT-4 doing much better at the law, at bar exams for law than GPT-3.5 and a lot of different areas that it excels at in a very measurable way. And I think that these costs might be one of the reasons Google has yet to build an AI that directly just integrates with its search engine, right?

4:33Like they've released and they've announced BARD and that BARD is going to be doing some cool things with AI inside of Google search. But to date, it's still not there. It's still not 100 % released. And a lot of people are saying it's just due to the sheer cost of this. So Dylan Patel, he's a chief analyst at a semiconductor research firm called Semi Analysis. And he estimated that a single chat with ChatGPT could cost you 1000 times as much as a simple Google search. So earlier, right, I said 10, 100 times, there's a lot of different estimates that are going around right now. But that 1000 times cost comes as we're looking at these new GPT for responses that we're getting that are much higher quality, but they also, they also use a lot more computational power.

5:19So in a bunch of different recent reports on AI, even the Biden administration is kind of weighing in from a government perspective on this issue. And they specifically, you know, highlighted the cost of generative AI as a national concern. So in a blog, or and I guess in a release, they wrote that the technology is expected to dramatically increase computational demand and the associated environmental impacts, and that there's urgent need to design more sustainable systems. So from the government right now, obviously, we have a left leaning government in power in America right now, who is quite focused on environmental impacts is that as part of their platform.

5:57So you can always kind of expect when they when they make comments on what is kind of cutting edge and happening today, they're going to comment with the with the plot, you know, how it relates to the platform they have and what they're focusing on. So environmental impacts is something that's very important to them. And so that is what they're kind of bringing up there. So I will say there is some hope, though, for new new coming AI companies and a lot of other people, there's been a lot of really impressive AI models released, meta released one that is quite famous, which is the Lambda model that, you know, famously, some researchers at Stanford were essentially able to clone ChatGPT for, for around$600.

6:37And the computational cost was a lot lower on that. Now that being said, I feel like it is kind of, I mean, cheating, I don't know. Like, they used responses that ChatGPT gave, which had already been like trained and fine tuned. So it's, it's in a way they're kind of taking some of the work that OpenAI ChatGPT did to create a model that could come up with those responses. So that's for the data that they fed into it. In addition, it's against OpenAI's terms of service to, you know, create a competitor based off of outputs from their own model makes sense. But I just want to bring that up because, you know, in all the talk of it being cheaper, it's not necessarily that easy.

7:15But that being said, these costs are coming down significantly. And so I do think that there's ways for newcomers to come into the market and create some still very powerful AI models. And I do believe that technology will continue to come out that is going to lower the cost. Now, that being said, one of the big costs that a lot of people talk about is just the infrastructure to run these models. So the GPUs required, Elon Musk famously tweeted or said in a, I believe in a press briefing or something that that these GPUs and these chips were harder to get than drugs right now. And he recently bought 10 ,000 GPUs for his new AI startup that he is stealth mode kind of building.

7:56And so I think that Silicon Valley really came to originally dominate the internet and all of that by offering things like Google search and email and social media, right, we're seeing from Facebook and everyone else for free or for losing money initially, but inevitably they turn it around, they start making, you know, some pretty big profits and they go public. And I mean, even companies like Twitter pre-acquisition by Elon Musk had always lost money and like it IPO'd, but it was never a profitable company. And so I think, you know, consumers are kind of used to this, but I don't know how sustainable it is in a world of AI where these costs are going up 10, 100, or 1000 times what their predecessor was when you're looking at the difference between just doing a Google search and getting a result from, you know, GPT-4 that's very in-depth and covers everything you're looking at.

8:50So it's going to be interesting to see like how long these companies can actually sustain these added costs. I would say it's no accident that all of these companies building and leading AI models are either like they're the largest cloud computing providers, or they've partnered with them or very closely partnered with them, as you're seeing with Google and Microsoft. And, you know, Anthropic has partnered with Google and OpenAI has partnered with Microsoft. And I think that this is something that we're going to see a trend in because just the costs are so heavy that you really need to have an in there in order to cut these down and be a viable business.

9:28Companies that buy those AI tools coming from those companies don't realize essentially they're being locked into a very heavily subsidized service that costs a lot more than what they're currently paying. That's something that Clem DeLangu, who's the CEO of Hugging Faces, recently said. So this is kind of interesting. Sam Altman also recently kind of alluded to this in his congressional hearing that he had. He said, I think Senator Josh Ossoff, a Democrat from Georgia, was warning that AI, well, he was kind of concerned that it was going to become addictive, chat GPT was going to become addictive or harmful for kids.

10:10In my perspective, I think, you know, looking at tools like TikTok, I see a lot more potential for harm there, perhaps maybe more addictive than chat gpt um but in any case i guess it's an interesting concept to think about and we should be open-minded in any case uh on this topic sam altman was responding to that and he said we try to design systems that do not maximize for engagement in fact we're short on gpus the less people that use their product the better so he's kind of alluding to the fact that it is really hard to get gpus you've seen it with open ai and chat gpt when you're trying to use it and it crashes, it only gets you through half of a response or an article.

10:46It's kind of frustrating. So the expense of these AI models is really, really high. And in addition to that, just the cost of hiring some of the talent to work at your company, from some of the financial filings of these companies, we're seeing star AI scientists being paid up to$5 million a year for their help in projects like OpenAI and Bing and Anthropics Cloud. So I think this is definitely something that's very expensive. I think it's something that we really have to think about what the cost is and how sustainable this is, right? Like if we go ahead and integrate ChatGPT into every single business vertical at the moment and then perhaps it has some monumental shifts, people are laid off from their jobs and then all of a sudden, two years down the road, the costs of running these AI models, everyone can't subsidize anymore, and they go up, you know, 10x or 100x, you know, what would the what would the implications of that be that they would be severe, no doubt.

11:48So I think this is something we're looking at. And I think that the idea or the goal is that right now people are willing to burn money with the expectation not only of, you know, taking market share, which is kind of in the traditional route of this loss leader, kind of idea or like strategy with tech, but I think they're doing this with the expectation in the hope that future models, they will be able to optimize and train and run for a lot less money. And I think with that assumption, this very well could be a sustainable strategy. But if that doesn't come to fruition, there would be some, you know, some serious consequences and a lot of a lot of serious costs, I think that would be passed on to the economy.

12:26So it's kind of interesting, the CEO of D matrix is a startup working on more efficient chips for AI. He said, this is not a sustainable equation for the democratization of wide availability of generative AI, the economy, or the environment. So there are a lot of people raising, you know, that are very well informed into this area that are raising these kind of red flags. Google back in February, like I was talking about earlier, said that it was going to initially run a lightweight version of their Lambda language, because essentially it required significantly less computing power, enabling them to scale to more users aka it's a lot cheaper um and they probably couldn't afford how it was um and you know it was interesting because just the fact that they cut the cost on that um and gave us a essentially worse version of bard um and made it so that it messed up some of its basic facts on launch and it cost them 100 billion dollars in stock valuation until you know they're able to get their big ai conference back together and kind of hype their stock up again so So I think that it's going to be interesting to see what happens.

13:28Something that Google also has recently announced is a new chatbot called Sparrow, which was essentially designed by the DeepMind, which is an AI subsidiary they purchased a while ago. And the purpose of Sparrow is to search and to cite all of the sources that it comes up with. So I think it'll be deeply integrated with their search with Bard and probably what Bard's doing there and then being able to cite all of those with the goal of reducing, you know, misinformation that it spits out. So I think this is going to be really interesting. A lot of people have been saying that essentially this cost is holding us back right now, which is a whole other issue I think that is really important to highlight.

14:05And that is the fact that right now, we may not be seeing the general public is not getting the best version of AI because the best versions are too expensive right now. But as the cost of computation comes down, I think we're going to see a broader rollout of even more powerful AIs. And if no new innovation, which as you see in AI, there's new innovations every single day, new companies, integrations, and breakthroughs. But if we saw no new innovations, aside from the fact that we were able to, you know, get energy costs down, and we were able to get more effective chips and GPUs and all of that, I think we all we would be able to see quite a large rollout of even more powerful AI.

14:44And so So because of that, I'm quite bullish, you could say, on the advancements that AI is going to continue to make. I don't think that the progress is going to slow down significantly because there's so many different variables that as they come to fruition, this is going to just accelerate. So I think all of this is going to make some big changes. And I think that a lot of critics also note that general AI also comes with costs to society. So someone recently said, all this processing has implications for greenhouse gas emissions. That was a, you know, a dean at Tufts University Fletcher School.

15:24So, you know, people have, they're different, people have different criticisms, definitely, of the downsides of this. Some people are saying based on an estimated chat GPT usage and computing needs, Casper Gross Albin Lutzviggen, he estimated that it might have used as much electricity in January alone as 175 ,000 people, which is around the equivalent of a midsize city. and obviously January was one thing, but the usage of this has scaled. Companies integrating this has scaled. So there are costs associated with this. And I think what's going to be interesting to follow is if this is going to be a bubble that is not popped by perhaps the AI progressing less, but eventually when people come to the realization that these things do cost a lot of money and someone's going to have to foot that bill, whether that's the company or the consumer, the cost will have to be paid unless we can find a way to dramatically cut these down.

16:20So this is going to be an area we're going to have to follow very closely. And I think we're going to see a lot of really incredible advancements in the way of cost cutting, faster GPUs, more effective usage there. And it's going to be interesting to see who the big players are in putting that forward.