Gemini Rising: Google's Next-Gen AI Training to Outpace GPT-4 by 5x

0:00I think the first thing to talk about here is that in the very rapidly evolving landscape of LLMS, Google and OpenAI have of course emerged as the main two titans here. I mean, we have people like Microsoft that obviously play heavily in this field, but right now they're outsourcing a lot of that to OpenAI. So, you know, at the end of the day, it's kind of Google and OpenAI. There's a couple others. These are the two big ones. But each of them have their own kind of distinct advantages and strategies. So a few years ago, Google released the MENA model, which temporarily was, you know, the winner of the world's best language model.

0:31So that was back when we were comparing it with OpenAI's GPT-2. So Mina actually had 1.7 times greater model capacity, and it was trained on 8.5 times more data back then. And this is actually pre-COVID. So Google engineer Noam Shazer, he had, you know, a prophetic internal memo that said, quote, Mina eats the world. And it kind of highlighted the increasing integration of language models into daily life and also kind of the dominance in globally deployed flops, which is floating point operations per second. it anyways shazer's insights which were pretty i'd say they're largely overlooked at the time i think they were um very big to the world's kind of later realization upon you know the release of open ai's uh gpt that's when all of a sudden people are like looking back and saying oh dang he probably was on to something so i would say google's moment in the sun was brief because just a few months later, OpenAI unveiled Jupyte 3, which was then kind of passing Amina with more than 75 times the parameters, and then it had, or 65 times the parameters, I think it had 60 times the token count, and it had 4 ,000 times more flops.

1:44So I think the gap in performance between the two models was very monumental, and despite the apparent setback, I think Google is far from being counted out, right? We've seen them come out with Google Bard, they're obviously a contender in the field fighting here. But according to internal projections, Google's ongoing efforts, especially with its Gemini models, are set to outperform the total pre-training flops of GPT-4 by 5x by the year's end. This is absolutely massive. So I think this path is also clear to exceed this by 20 times by the end of the following year. But then again, of course, OpenAI is going to have to come up with something by then.

2:24So I wouldn't really count on. I wouldn't put too much speculation into, you know, what their latest things are going to do in a year's time. So this is all thanks to Google's really robust infrastructure build out. So the question remains, right, is whether Google is going to deploy these models publicly without compromising their creative abilities or their existing business model. But I think, you know, it's not just about Google and OpenAI. I think the broader landscape reveals a very significant divide between the GPU rich and the GPU poor. So essentially, these are factions in the machine learning world.

2:57The former group, including labs at OpenAI, Google, and other tech giants like Meta have essentially access to substantial computing resources. This is like, in the case of Google, this is something they built up over long term for a lot of their projects. And in the case of others, it's something that they may maybe potentially recently acquired, right but we're seeing a big pinch right now with gpu shortages from nvidia we're seeing nvidia go to all-time highs passing a trillion dollar valuation um and so it's kind of this moment where if you have it you have it if you don't you don't you've got to try to build it up i mean even elon musk at this point i think has just bought and is about to put online like 250 million dollars worth of gpus uh from nvidia to help training i think they're a100s if i'm not mistaken um but like it's kind of hard to to compete with that right 250 million dollars a company like tesla can do but definitely not for everyone so i think this kind of computational muscle has even become a bragging point among a lot of top machine learning researchers and even a recruiting strategy for companies so on the other hand the gpu poor you could call them is essentially compromising startups and open source researchers and these people are grappling with essentially limited resources, right?

4:08They're often kind of squandering valuable time on impractical projects. And I think that these constraints significantly limit their ability to compete in a landscape that is rapidly scaling up. So in the midst of all of this, NVIDIA, with its really extensive offering in cloud services and supercomputers, has become a very critical player. Effectively, it's just eating the lunch of more resource-constrained companies like Hugging Faces, Databricks and Together. And these firms are hard pressed to catch up to NVIDIA's pace, essentially lacking both the computational resources and the capital to substantially invest in the infrastructure.

4:46So it's kind of this two-edged sword and NVIDIA is really, really pulling ahead. So essentially many technology firms and startups are funneling capital into NVIDIA's bank account, setting up an industry dynamic where NVIDIA holds a significant power stake. So while NVIDIA's dominance appears, you know, relatively unshakable, Google with its rich compute resources could potentially be the industry's white knight. So Google's prowess isn't confined to its language model. It extends to its infrastructure, right? This is something that Google has been building up for a very long time for many of its different core competencies in its business.

5:25So particularly its AI optimizer tensor processing units, TPUs, like TPU v5, are also known as Viperfish. So Google's infrastructure efficiency could be its trump card in this whole situation, essentially allowing it to rapidly scale up operations. So therefore, while the spotlight often shifts between individual advancements and language models, I think the real game changer might be infrastructure with Google's really relentless pace in building out its AI capabilities. I think it may very well reset the playing field, challenging not just open AI, but also potentially loosening NVIDIA's type grip on the market, right?

6:06Because if Google can come out and essentially offer companies the ability to train models with much, much less compute, all of a sudden it diminishes the need for everyone to go out and buy hundreds of millions of dollars in NVIDIA chips. Essentially, we're able to just train it for this a lot more efficient. And this really should be a positive to the economy. This should be something everyone is supportive of because it's great for the environment. It's great for energy consumption. It's great for a lot of different reasons. to be able to be more efficient and so it's going to be really interesting if google is not only kind of stealing a little bit of thunder from nvidia with some of these uh tpus but also by using it it's like internally if they're able to essentially displace open ai by now creating bigger models better models that were trained more efficiently and it's just something that open ai would have to spend extremely heavy to try to to try to catch up so very interesting space Google may have one less trump card up its sleeve and that is what we're going to be following into the future