AI Evolution: From Google to Nature - The Journey of a Transformer Paper Author

0:00The holidays are here and that means it's the most wonderful time of the year to save with Rakuten. Use Rakuten to stack cashback at your favorite stores on top of holiday sales. That's savings on savings. With Racketing, you get cash back on gifts for everyone on your list. From toys for the kids, to kitchen gear for the person who loves to cook, to electronics for everyone. You can even save on something for yourself. Just shop the stores you love and cash back is automatically added to your account. And you can get paid with gift cards, PayPal or check. or eligible American Express card members can even choose to earn membership rewards points instead of cashback.

0:41It's truly a no-brainer. Join for free today and get a new member bonus after minimum qualifying purchases. Just go to Rakuten.com, download the app, or install the browser extension. That's R-A-K-U-T-E-N. Terms and conditions apply. What can 160 years of experience teach you about the future? When it comes to protecting what matters, Pacific Life provides life insurance, retirement income, and employee benefits for people and businesses building a more confident tomorrow. Strategies rooted in strength and backed by experience. Ask a financial professional how Pacific Life can help you today. Pacific Life Insurance Company, Omaha, Nebraska, and in New York.

1:24Pacific Life and Annuity, Phoenix, Arizona. Today on the podcast, we are talking about a new startup called Sakana AI out of Japan that wants to build nature-inspired artificial intelligence. This was founded by former Google AI luminaries Leon Jones and David Ha. And Sakana AI essentially promises to deliver a revolutionary approach to foundational AI models. So today on the podcast, we're going to be diving into what they are building and why we think that this is relatively important in the industry today. So Jones isn't new to revolutionary ideas. He actually co-authored the 2017 groundbreaking Google research paper, attention is all you need.

2:03It's kind of funny because I swear every single person that co-authored that paper has gone on to found some of the biggest AI companies. I'm not sure if I mean obviously they were like very ahead of their time and came up with a very interesting concept but I think beyond that just the brand name and the name recognition of that paper and how important it is made it really easy for all of them to go and you know get venture funding for their uh their companies right if they were that early on that and were able to come up that bit of revolutionary technology they're obviously deeply ingrained in the space and poised to to do some really great things with research and tech so it's kind of funny to see or it's exciting to see all of these guys come out and have really awesome AI companies.

2:42So Jones was working at Google for 12 years. He just left and he is now the CTO of Sakana AI. And on the other side, we have Ha, who is, you know, essentially is the CEO. And he previously spearheaded Google's AI research initiatives in Japan. He also helmed research at AI's Upstart Stability AI. both Jones and Haw are on a mission to conceive AI models rooted in the insight drawn from natural systems now this is awesome both of these guys have a ton of experience at big AI companies making big AI plays and so I'm here for it to see what they're going to be building but essentially Sakana itself translates to fish in Japan and that is essentially what their philosophy is So the duo envisions AI models that much like a school of fish are individual units synchronizing seamlessly to operate as one unified entity.

3:35I'm drawing from the intelligence exhibited by nature and fundamentals such as evolution and emergence. They are committed to essentially devising AI solutions that are adaptable rather than static and inflexible. Now, one thing I do want to say about this, you know, like having the whole school, the school of fish concept or whatever, this isn't entirely new. And it's not like, oh, my gosh, these are the first people to do this. I'm sure they're going to do a lot of really innovative things. And I'll cover some more things that they're doing here that's interesting. But what I did want to bring up is that ChatGBT is actually doing this in a way, probably on not such a granular scale that they're about to go into.

4:12But I do think this concept is really interesting. And so I think that they're actually in the right direction. because essentially what was there was a bunch of leaks recently for model weights of chat chpt and we were able to get a lot of insights into what they're doing but essentially in that leak we found out that chat chpt is essentially segmented into 16 different professionals and um so it's like there's 16 mini models in chat chpt there's one model when you ask a question it determines which of the 16 models could respond best that's why sometimes i feel like when i ask questions in different ways like the answer like it was like really variably different um and i think it's because it may have switched over to a new one of those like professionals or experts but anyways that's that's a theory i'm not 100 sure about in any case um that's exactly how chadgbt is working they they essentially determine which is 16 they have an expert that responds to you and they have it segmented out this i think could be the solution if you go a little bit further i'm hoping with chai chippie t they go a little bit for chippie t5 they go a little bit further um because this could be the solution to like chai chippie not being good at math where essentially um essentially it can integrate multiple of these experts into one thing right so it's like it's going to have the logic and reasoning expert that's really good at math mixed with one that's good at like i don't know english or something because sometimes it's tricky in a sense that uh it feels like you're getting one of the experts and not the other right now but in any case I think that could fix a lot of the problems but it looks like that's exactly what Sakana is kind of aiming to do here so I think this could be an antidote to the challenges like exorbitant costs and looming security threats posed by current AI models models that consume mass amounts of data and computing resources like ChaiJPT are obviously incredibly expensive and this also might be a way to cut down on those costs so rather than merely expanding transformer models Jones and Ha are it means to kind of architect innovative leaner techniques, right?

6:11They're not going to make these models that cost millions and millions of dollars, although they may still cost millions of dollars, but you know, I think OpenAI spent like over$540 million last year, something crazy. And so obviously, they probably are going to have that kind of arsenal of cash. So they're going to be doing this lean, which I think is good. It's going to use a lot less resources, making a lot of smaller models. So this will be interesting to see how this kind of leaner technique plays out. So without going into the nitty gritty, essentially they are building, they're looking at the possibility of smaller interconnected models working in tandem, sort of like a flock of birds or school of fish to tackle challenges.

6:49This methodology really promises to be more sustainable and secure than models centralized around one massive data set. I also think this is a wise move if we're seeing a lot of lawsuits come out right now, right? New York Times is suing or looking at suing OpenAI for using its data. A lot of other people would follow suit if they won. And so imagine, if you will, where you have like, you know, a thousand mini AI models that are all using different data sets, doing different things. and if for some reason you have a massive lawsuit in one of these thousand ai models you could just remove that model um or though preferably you could figure out a way to just retrain it but after removing that thing from the data set you don't have to retrain the entire thing um it's really hard to extract one data set from um uh from like chai gbt for example because you've already trained the whole thing and that was included but now it's like well we have a thousand so we just have to like retrain a thousandth of our model excluding that data set so i think in that regard too this also could be a really good play to kind of segment out the ai models into a lot of smaller kind of micro models if you will so i think this might be a good play in that regard um and i think that the decision essentially was to establish this company in tokyo which is really cool um not a lot of ai companies that are making headline news right now we're out of tokyo a lot of new york and san francisco so i like to see a little diversification and location here while several former Google AI employees have congregated to global AI epicenters like Silicon Valley Jones and Hall have charted a different path they say that Tokyo with its robust technical backbone pool of educated professionals and thriving research holds some unique appeal so this is one other thing that I will say that I like about this move beyond just like oh it's a new place other than Silicon Valley.

8:43I really think that if we want to have a very global and robust and diverse AI ecosystem, we need to do this. We need to spread out. We can't just have everyone from everywhere around the world all congregate in Silicon Valley inside of what is sort of a tech think bubble where a lot of people all think the same. There is so many different places around the world with so many different ways of thinking. And I do think it's very healthy to have a lot of different perspective. Now, I know most of my listeners are from California, but I'm sure you guys, you all see it too, right? There is a lot of people think the same way inside of their own bubbles and circles.

9:20And so I think spreading it out gives you a lot of diversity, which is really, really powerful and a great thing to see. So I think that the city has a lot of international expertise combined with its potential to mold AI solutions for non-Western context. I think that holds a lot of promise. A lot of different Asian cultures have different ways of thinking or solving problems that has been traditionally really helpful in a global system of trade. And now I think it's going to be good to add this is an AI layer here as well. So I think it's no secret that the AI domain is really saturated right now.

9:59There's a lot of companies coming into this space and playing in this. We have Google, Microsoft that are really kind of fighting companies and startups like Anthropic and Cohere and OpenAI and all that kind of stuff. But Jones and Haad discern a, or essentially they see what they call a concerning trend, which is the overpowering commercial drive that could be dampening genuine innovation. So their ethos is to reignite the spirit of unbridled exploration. I know that's kind of generic, so cool, whatever. I'm not gonna put much credence in that until I actually see what they produce. But in any case, I think their unique trajectory of Secchaon AI really resonates with researchers who are looking for a shift from really, I don't know, commercial rigidity to something a little bit more exciting and interesting.

10:43So regardless of if that headline there actually means anything, I think it does a good job in getting some of the top talent to want to work at your company. So in that regard, you know, tip of the hat, this is you got to do what you got to do. And I think that's a great move. So the company has already onboarded an academic scholar and is looking for more visionaries passionate about foundational research. I think their journey is really about building something innovative. You know, I think given its leadership's strong foundation and their innovative approach, I think it might be able to be a really big disruptor in the AI realm and introduce some foundational principles that, you know, are going to shape the way a lot of things are happening in AI.

11:24If this approach that they are working on proves to be true, I think we would see big companies like OpenAI and others follow suit and take more of this nature-based approach where essentially you're segmenting all your AIs into many, many micro AIs, and it's much, much more efficient and perhaps better protected against lawsuits and it actually gives better responses. So this is what we're going to be left to see as they continue developing.