Jeff Dean on TPUs, AI Research, and Funding

17:54Jeff Dean:They're like the electricity or the plumbing of the Internet age. Absolutely. And they specifically cite some foundational elements. Things like TCPIP, the protocol that defines how data moves across the Internet. That was designed in open academic and government settings. They cite the architecture of advanced RISC processors, which pioneered the efficient, streamlined instruction sets that specialized chips like TPUs now mimic. And then there's the really powerful anecdote that connects directly to the company's own Genesis story. Right. They point to the specific funding mechanism that led to one of Google's earliest and most central innovations, which was the Stanford Digital Library Project.

18:35Now, this was a publicly funded initiative, and crucially, it provided the money for the original graduate student research project that eventually became the PageRank algorithm at Stanford University.

18:46Jeff Dean:So you're saying PageRank, the algorithm that basically defined modern search and launched the modern Internet economy, was the direct output of a publicly funded academic research project. That's exactly right. The return on investment for that initial relatively small academic grant is, it's just astronomical. You can't even calculate it. It's the ultimate evidence for the societal returns of funding basic research. It is. And this history extends deep into the AI revolution itself. The deep learning revolution that we are all experiencing today isn't some recent invention. It is built on fundamental academic research that is 30 to 40 years old.

19:23Jeff Dean:You're talking about the core inventions of neural networks and backpropagation, which were largely developed and theorized back in the 1980s and 1990s. The concepts were mathematically sound, but they just sort of sat dormant for decades. They did. They were part of the connectionism movement, which was often dismissed as being impractical during the dominance of symbolic AI. The mathematical groundwork was all there, but what was missing was the data and the compute. The very things the TPU was later designed to provide efficiently. Exactly. The academic seeds waited for decades for the technical infrastructure to finally catch up.

19:56Jeff Dean:That lag 30 to 40 years between the initial academic insight and the planetary scale commercial application, that's a kind of a terrifying thought when you consider the current rate of technological acceleration. And that realization leads directly to the core advocacy point from the source material, which is that a vibrant academic research ecosystem is absolutely essential. These institutions, these universities, are where the early stage creative, often counterintuitive and riskier ideas get explored without any immediate commercial pressure. These are the 30 year seeds. These are the ideas that lead to the major breakthroughs.

20:33They are the 30 year seeds that grow into global industries.

20:35Jeff Dean:But OK, let me challenge that for a second. If the returns are so vast, why should taxpayer money or public funding organizations like the NSF bear the full risk? Shouldn't these multi-trillion dollar companies who are now profiting from those ideas be the ones paying for the next generation of foundational research? And that's a completely fair challenge. The reality is that fundamental research often moves too slowly and is far too high risk for even the largest corporate R &D divisions to fully fund without a really clear path to monetization. What's more, the outputs of public academic research, published papers, open source code, they are inherently public goods.

21:14When a university publishes an idea, everyone benefits, which maximizes the societal return.

21:20Jeff Dean:So the argument is that we need a robust public funding model. The chief scientist's point is that society needs to ensure a robust funding model for this basic research, because the long-term returns are just far too large and too beneficial to be left purely to market forces. So before we can even talk about funding the next product, we absolutely must ensure we're funding the next foundation. So taking that point about funding the foundation a bit further, the discussion naturally moves to the need for alternative funding mechanisms for more applied AI research. If the traditional short-term grants just aren't adequate for the scale and the timeline you need for modern cross-disciplinary AI challenges, then what organizational model is best suited to bridge that gap.

22:00Jeff Dean:The gap between pure academic research and immediate, large-scale societal application. You need a model that fosters system building, not just paper writing. And the source details a very specific example that's designed to solve exactly this problem, the Lott Institute model. Okay, tell us about the structure of this nonprofit. It sounds like it's intentionally designed to avoid the short-term quarterly pressures of commercial work, while at the same time providing more sustained resources and a clearer timeline than a traditional university grant. It's a very deliberate, highly structured entity.

22:34It operates as a non-profit, a 501c3 in the U.S. structure, and it specifically raises money from successful technologists, so people who have already benefited immensely from the academic foundations we just talked about.

22:45Jeff Dean:Paying it forward. In a sense, yes, and its core function is to run a moonshot grant program that is dedicated to funding full research labs, not just individual graduate students. And what defines one of these labs? What makes it different? They are dedicated to funding research labs over a very targeted three to five year period. And these labs, they're structured to be sizable and interdisciplinary. You're talking about three to five principal investigators or PIs and typically 30 to 50 PhD students, postdocs and research engineers all working collaboratively under one roof. This concentration of talent combined with that clear time horizon is really the engine of the model.

23:25Jeff Dean:And that structure, the size, the interdisciplinary nature, and especially that time horizon, that seems to be key. Why is that three to five year window considered the delightful sweet spot for these ambitious research goals? Well, if the time horizon is too short, say a typical two year grant, just can't even conceive of doing anything truly ambitious that requires building complex systems or negotiating real-world agreements and deploying systems in testing environments. Two years is enough time to write a paper. It is not enough time to build a robust production-scale system. And if it's too long, say, 10 years?

23:59If it's too distant, the project just loses its urgency. The goals become too abstract, and the personnel might cycle out before you ever see the payoff. Three to five years is described as not so distant that it won't have impact, but not so short that it prevents ambitious goals. It forces tangible system-building deliverables while allowing enough time for the team to navigate all the inevitable technical policy and grungy hurdles.

24:23Jeff Dean:And this approach, it shifts the focus pretty dramatically away from the distant theoretical promise of AGI, which seems to dominate the conversation right now, and toward these achievable, high-impact societal goals. Absolutely. The research in these models is explicitly targeted toward measurable impact in areas like accelerating scientific progress, improving health care outcomes, facilitating job reskilling in vulnerable sectors, enhancing civic discourse. The focus is on tackling solvable problems using technology that often already exists or is very near fruition. The goal is to eliminate human drudgery or bureaucratic friction.

25:00Jeff Dean:And the source highlighted the application of AI in health with a real sense of personal passion, setting this incredibly audacious aspirational goal for society. The goal is profound because it links every single data point. The aspiration is, how can we as a society use every past decision that's been made in health to inform every future decision? That level of continuous scaled learning, a true learning health system, it would fundamentally change diagnostics, treatment protocols, and disease prevention on a global scale. That sounds like a technical ideal, but the reality of healthcare data, I mean, it must introduce just massive hurdles.

25:36Jeff Dean:You can't just throw all the world's patient data into a single cluster and train a model. The privacy and regulatory concerns are just immense. And that brings us right to the first major obstacle. The technical hurdle, which is rooted in privacy. Because of strict regulations like IMPA in the U.S. and GDPR internationally, data cannot be moved easily, if at all. And this mandates a fundamental shift in how machine learning is even performed. Right. So instead of bringing all the data to a central model, you have to bring the model to the data. This requires solutions like federated learning.

26:11Jeff Dean:Can you explain how that works in practice? Because the mechanism seems key to addressing these privacy concerns. Federated learning really flips the traditional centralized ML model on its head. In an applied health setting, there are generally four key steps. Okay. First, model initialization. Central server trains an initial sort of simple version of the model architecture. Second, distribution to silos. This initial model is securely distributed to the data silos, so individual hospitals, clinics, or regional health systems. And the raw patient data stays securely behind their local firewalls.

26:42Jeff Dean:Never leaves the hospital. Never leaves. Third, local training. Each hospital trains the model on its own local patient data. So the model learns the specific nuances of that particular population. And fourth, and this is the most important part, secure aggregation. The hospitals do not send the raw patient data back. What do they send? They only send the aggregated updates or the model parameters. So just the mathematical changes the model underwent. And they send that back to the central server, often using techniques like secure aggregation or differential privacy to mask contributions from any individual patient.

27:16The central server then averages these updates to create a more robust, generalized global model, which is then redistributed for another round of local training.

27:25Jeff Dean:The crucial point is that the sensitive patient data never, ever leaves the safety of the local jurisdiction. It provides learning at a massive scale while respecting those privacy boundaries. Precisely. It's a technical solution to a legal and ethical problem. But privacy isn't the only hurdle. The source calls the second big challenge the grungy hurdle. That sounds like the messy reality of data collection in the wild. It is the brutal reality of data heterogeneity. I mean, just think about the countless different ways medical records are kept. One system uses ICD-9 codes. Another one uses ICD-10.

28:00Some records are meticulously structured, while others are dominated by free text, unstructured doctor's notes.

28:07Jeff Dean:It's a complete mess. It's a mess. And getting this mountain of disparate, non-standardized data into a usable, structured form for training a global model is this immense, non-glamorous, but absolutely necessary undertaking. It requires a heavy investment in data cleaning and standardization tools. And finally, you have the ultimate complexity, which is the policy hurdle. The legal and regulatory complexities are incredibly high, and they change across every single jurisdiction. You have different consent laws in different states, let alone across national borders and international regulatory regimes.

28:41And this legal and policy landscape is constantly shifting, and it's often contradictory.

28:46Jeff Dean:So this is why you need that kind of lab structure. This complex environment is the exact justification for why the Law Institute structure is so vital. You cannot solve this problem with a small team of ML specialists. You need groups with mixed expertise, ML specialists, high-performance computer systems builders, and specialized legal, policy, and regulatory experts all working in tandem on a multi-year project to achieve any kind of real-world deployment. It sounds like the technical challenge is maybe even easier than the policy challenge in this context. Often, yes. The policy work, the negotiations, the regulatory mapping, the trust building, that can take far, far longer than the model training itself.

29:25Jeff Dean:So this incredible internal investment building, custom silicon, managing training jobs across multiple cities, tackling these global health challenges, it all raises a fundamental question about the boundaries of corporate research. How does a company with that level of internal advantage manage its relationship with the broader open academic ecosystem that it so fundamentally relies upon? This is a really complex balancing act, especially in the current highly competitive dynamic of the AI race. The chief scientist explains that publishing is not simply a binary choice. You know, we publish it or we don't.

30:00It's actually a nuanced, deliberate continuum based on product cycle maturity, competitive advantage and the nature of the research itself.

30:07Jeff Dean:OK, so let's break down the three defined points on this continuum, starting with type one, the earliest stage open research contribution. Right. Type 1 is open early stage research. And this involves publishing new earlier stage model architectures and ideas that haven't necessarily been proven out or experimented with at a massive scale. The purpose here is purely to contribute to the global research dialogue and to invite external validation and exploration. This is often high risk, high reward architectural exploration that really needs external partners to test and refine. Do we have a specific example of this kind of speculative early stage paper?

30:43We do. The source cites a paper that was published on a hybrid transformer and recurrent model called Titan. Now, the core problem that Titan is trying to solve is the computational complexity of the transformer's attention mechanism.

30:56Jeff Dean:The famous n-squared bottleneck. That's the one. It scales quadratically with the length of the input sequence, which means that making the context length longer is extremely expensive. Right. So Titan proposes a solution using recurrence relations, but instead of applying recurrence on individual tokens, it applies recurrence to chunks of tokens, say a block of 512 tokens at a time. The model learns to compress those 512 tokens into a summary or a memory vector, and then it uses recurrence steps on the sequence of those compressed chunks. So it's a hybrid approach. It is. It's aiming to achieve significantly longer context windows, the ability to reason over much larger documents with far greater efficiency than standard models.

31:39And the crucial detail here is that the chief scientist explicitly notes that Titan is not currently used in the Gemini models.

31:45Jeff Dean:Oh, interesting. But it is an interesting architectural idea for future exploration. And by publishing it, they're making a high value intellectual contribution back to the academic ecosystem. Okay. That clearly shows a commitment to the fundamental research ecosystem. Now let's move to type two, which sounds a lot more focused on commercial strategy. It is. Type two is delayed publishing, prioritizing the product advantage. In this very common scenario, the innovation is developed internally, it's rapidly deployed into a consumer product to gain a market advantage, and then the science detailing how it works is published much later.

32:20Jeff Dean:And the most salient examples here come from their deep work in computational photography. That's something that has just radically changed the capabilities of mobile phone cameras. It is the perfect illustration of applied ML innovation. I mean, think of features like Night Sight, which uses deep learning to synthesize multiple short exposures into this vivid single low light photo. Or astrophotography mode. Or astrophotography, or the popular magic eraser, which relies on these sophisticated semantic segmentation and in-painting models to just seamlessly erase a person or an object that wandered into your photo.

32:57Jeff Dean:That ability to make something feel like user-friendly magic while relying on this incredibly complex ML underneath, that's really the hallmark of great product research. And the process is very carefully timed. The feature is developed internally, it's rigorously tested, and then it's deployed in the next flagship phone, the Pixel N Plus One. They maintain the product lead for a set period, usually 6 to 12 months, and then they submit a SIGGRAPH paper detailing the precise underlying innovations. So they get the best of both worlds. It ensures the innovation eventually enters the public domain, and academia can build on it, but the company reaps the initial commercial advantage.

33:32It's a way of balancing competition and contribution pretty effectively.

33:36Jeff Dean:And finally, we have type 3, which is just the necessary reality of the current competitive dynamic, the secret sauce that has to stay internal. Given the scale of the investment, the custom TPUs, the pathway systems, the massive datasets, there are certain specific architectural details that just have to remain internal and proprietary. The source explicitly mentions that they do not publish the final secret architecture inside the massive Gemini model. In a global race for large model leadership, key differentiating secrets must be protected. And beyond just external publications, the sheer scale of the internal research effort, we're talking thousands of PhDs and engineers, that means they have to manage their own closed ecosystem of knowledge sharing just to accelerate the pace of invention.

34:22They do. They host an enormous internal research conference, which the source cites as having about 6 ,000 attendees annually. This is essentially an internal, massively scaled version of a conference like NeurIPS. Wow. And the observation cited by many PhD students is that papers presented at this internal conference often feel a year ahead of the work that's presented at external public conferences.

34:44Jeff Dean:Why the perceived gap? I mean, is the work genuinely a year ahead, or is it just a matter of presentation and maturity? It's primarily a matter of the bar for maturity. To get a paper accepted at a top-tier external conference like NeurIPS or ICML, The work has to be meticulous, fully proven out, and quite fully baked. And that requires months of polishing, validation, and writing. Which introduces a significant lag time into the whole process. Exactly. The internal conference, however, it accepts a whole range of maturity. They specifically utilize lightning sessions for cool, early-stage results that are just too fresh, too speculative, or too incomplete to be fully proven or written up in a formal paper.

35:25And this allows for a very fast circulation of ideas among internal colleagues without the pressure of external academic rigor or the fear of premature judgment.

35:34Jeff Dean:It creates a rapid-fire feedback loop for experimental ideas. A massive internal competitive advantage that just accelerates their R &D cycle. It really shows that even with a massive vertical stack and trillion parameter models, the ultimate engine of progress is still the human element. It's the creative freedom to share speculative ideas quickly and iterate without the paralysis of perfection. Hashtag tag outro. So we've taken the full journey today, really tracing the core pillars of modern AI innovation. We started with that fundamental crisis back in 2013, the realization that standard compute just couldn't scale the workload.

36:08Jeff Dean:And that led to the specialized hardware solution, the TPU, built for maximum energy efficiency and that vertical co-design. We followed that hardware all the way up to stack to Pathways, which manages the monumental feat of providing the illusion of a single system image across thousands of geographically dispersed chips. It turns a massive, messy network into a seamless computational fabric. And this vertical integration is what allows for the current unprecedented scale of large model training. And then we synthesized that high-tech vertical stack with its necessary foundation. The debt that is owed to public, academic research, TCPIP, RISC processors, PageRank, and the very invention of neural networks all developed 30 or 40 years ago.

36:49And that realization led us to the crucial discussion of the future, the need to shift our funding models away from chasing hype and toward these practical, achievable AI moonshots. We identified that three to five year time horizon as the optimal window for ambitious cross-disciplinary teams to tackle major societal challenges like health, addressing all the technical, the grungy and the policy hurdles along the way.

37:12Jeff Dean:So what does this all mean for the immediate future? Well, the next era of impactful AI isn't solely defined by who announces the next biggest ship or the largest model. It is defined by highly specialized hardware that's built for efficiency, combined with a concerted, collaborative, and properly funded effort to solve real-world problems on a practical three - to five-year timeline. And it means we have to be willing to invest in the unproven today, knowing that the real returns may not arrive for decades. It means we have to remember that today's competitive advantage is tomorrow's public foundation.

37:43Jeff Dean:And that leads to our final provocative thought for you, the learner, to consider. If the foundational ideas for today's planetary-scale AI things like neural networks and backpropagation were 30 to 40 years in the making, and the most impactful near-future projects are now being defined by a maximum five-year timeline, what current early-stage research idea, right now sitting in an internal lightning session or a small-scale academic paper, will truly define the world in 2030? What small seed is being planted today that will require a future hardware crisis to solve 15 years from now? Something to mull over until our next deep dive.