Bridging AI Research and Real-World Impact — Dr. Petar Veličković, Senior Staff Research Scientist at Google DeepMind

0:00People who do data science and machine learning, we're often really excited just by seeing one number being bigger than another number, if that makes sense. Especially for people who are like, you know, classically academically trained like the two of us. There is a gap between seeing one number being higher and actually building something that people want to use.

0:26Welcome to Data & AI Mastery, the podcast where we bring you cutting-edge insights, practical advice, and inspiring stories from the leaders shaping the future of data and AI across the globe. I'm your host, Raoul Gabriel-Urma, founder of Cambridge Spark, the leader in transformational data and AI upskilling, career development, and progression. In each episode, I will be diving into real-world case studies of companies harnessing the power of AI to drive innovation, reduce costs, and create new business opportunities. So whether you are an aspiring data scientist, AI engineer, or seasoned executive, this show is designed to give you the tools and knowledge to stay ahead in a world where data is transforming every aspect of business.

1:10Stay ahead, stay inspired, stay masterful. Welcome to Data and AI Mastery.

1:20Hey Petar, how's it going? Hey Raoul, great to have a chance to speak with you once more. Everything is going just fine. How about you? Hey, really good. I'm looking forward to this conversation. You know, we met over a decade ago in Cambridge whilst both of us were doing a PhD and a lot has happened since, so I can't wait to deep dive in your story, Petar. Can't believe it's been a decade, yeah. Can't wait as well. Time flies.

1:58Before we get started, let's talk a bit about you and your story, because I think it's really inspiring. Can you tell us, you know, what did it take to be where you are? Trace a bit the history for us. I like to think about it as a sequence of very fortunate events, which all played a pivotal role in, well, either my growth as a scientist or in terms of the network I've built in research or in education and all the way to, you know, the collaborations I was able to form and the great downstream partnerships our products have spurred. In fact, I would probably count Cambridge Spark as one of the points along that journey.

2:42I'm sure we'll get a chance to talk about this as well. But yeah, I would say key points. I was born and raised in Belgrade in Serbia, where I had a chance to study at the Mathematical Grammar School, which is basically a high school offering undergraduate university level mathematics to students who exhibit certain STEM capabilities. I would say this was the first supercharge that allowed me to gain a solid grounding in STEM more generally. It became quickly apparent to me that it was computers and programming and algorithms that were most relevant to my skill set and what I was able to kind of absorb most easily compared to other things like maths or physics.

3:35And therefore, I decided I wanted to study computer science. The second lucky break happened when I was able to come to Trinity College, Cambridge to study for my undergraduate degree. And three years later, I guess, an even bigger lucky break happened when I was able to start my PhD in the same department where I've done my undergraduate. This was with Professor Pietro Leo, who fortunately happened to have a PhD funding at the exact time when I needed it. and then during the PhD I would say several very important things happened which shaped me as a researcher and as an educator on one side I've had the great pleasure to do several internships at the Montreal Institute of Learning Algorithms or MILA as it is known today with Joshua Bengio's team where we had a chance to plant the seeds of what is today attentional mechanisms in graph for presentation learning, which is one of the works that perhaps I'm best known for today.

4:41But while at the same time getting that proper big deep learning lab experience, I also had a chance to really gain a lot more confidence as an educator, primarily through working with you and your team at Cambridge Spark and Cambridge Coding Academy. This was basically on one side what gave me the confidence to get into more into education myself. and secondly also got to meet some really cool people which turned out to be really important connections for my future as well. And actually one of the connections I met through Cambridge Spark then connected me to some other folks at DeepMind who were then able to refer me when I wanted to apply to go work there when I was closing up my PhD.

5:29and fortunately I was able to get through the DeepMinds interview process and I started working there shortly after submitting my PhD in January 2019. And yeah, I've been at GDM ever since. I would say with the brief excursions back into education, I'm currently an affiliate lecturer at the University of Cambridge. So some years after I graduated, I decided to come back and take up teaching. And I currently teach a master's course in geometric deep learning there. So you're a senior staff research scientist at Google DeepMind. It's a long and impressive job title. So maybe to get us going, can you tell us what it involves?

6:14We meet at a dinner party. How do you explain the work that you do? The main aspect of what I work on is foundational research. So basically trying to understand what are the present limitations of artificial intelligence systems and then basically finding smart ways to work around those limitations so that one day down the line we can then take that research and convert it into something that can meaningfully impact the next generation of AI systems. this could apply to something like a chatbot like Gemini that probably you've had a chance to interact with in the past but also it involves a plethora of possible scientific and industrial and deployed artificial intelligence projects that DeepMind is involved in which I'm sure we'll have a chance to discuss during this conversation in more detail but yeah basically in a nutshell, it is working at the frontier of what we understand about modern artificial intelligence systems, trying to find ways to go even further beyond that, and basically trying to embed these takeaways wherever it might make the most sense to do so.

7:33All right, I can't wait to deep dive into those topics, Petra.

7:43so pedo let's uh let's talk through a bit about the work that you've done so the application of ai and um to get us started like maybe at a super high level can you talk us through what are like the last kind of couple of projects you've been involved at google deep mind and then then we'll deep dive into each one of them. When it comes to the actual deployed projects, it's maybe closely related to some other things people might know about me in recent times. So for example, we have deployed a graph machine learning system within Google Maps as the travel time predictor. So this is now, by the way, a system that is deployed worldwide.

8:29So billions of people interact with it regularly. when you open Google Maps and you say, I want to travel from A to B, myself in London, yourself in Barcelona, wherever you are, and the system will suggest to you the possible route you should take, and it will also recommend a travel time. So how much it estimates it will take you to travel in this particular route. That particular time estimate is produced by our model, which, as I said, it's a graph machine learning model, meaning that it takes into account not only the characteristics of the route itself, but also the broader topological structure of the road network around it, which basically can then account for things like unexpected rush hours, traffic jams, or things like that.

9:13And as a result, when we deployed this system, especially in certain metropolitan areas, it gave a much greater reduction in negative outcomes in in terms of predictions that were very incorrect compared to what the previous Google production baseline was able to do, especially in cities where traffic was notoriously harder to predict. So you can imagine some cities in India, cities in Australia, a few places in the United States as well. So there were particular areas where this kind of topological approach gave a very clear improvement. Peter, what does it take to build such a system? Can you talk us through like the scale, the team involved, like bring it to life for us?

9:57Right. I would say the team that actually ideates on these things usually tends to be a pretty lightweight team because fundamentally we start with a research idea, right? Like we want to validate whether a hypothesis even makes sense, right? So we will partner with a particular product team that will help us understand the environment and what we're dealing with. But usually it will be a relatively small team of, you know, some number of engineers and researchers teaming up together to try to figure out what is some simple way in which these kinds of methodologies could give you an uplift. And if you're fortunate enough, your hypothesis will be correct.

10:36And in this case, modeling the road network as a graph explicitly gives you tangible improvements on predicting the travel times on some particular data set that you've been given, right? Now, the problem is that many machine learning specialists do this and then they call it a day. They stop at that point, they publish their paper, and they're done, right? But the path to actual impact and deployment is usually much longer than this and requires a lot more considerations. So just because it works in the offline regime doesn't mean it will work well when you put it in a production system. You know, roads get built and demolished over time.

11:11The network evolves. The cars people drive evolve over the years, you know, like essentially or even what does it mean for a road to be good might evolve over time. In the past, you might only care about the shortest, fastest possible way to get somewhere. Nowadays, because of climate change, you might actually care about how do you get there in the most carbon neutral way or something like this, right? So sometimes you might even recommend someone to go on their bike to the location if this is within some certain parameters, even though travel time-wise it might not be the best choice, right?

11:47There's a lot more considerations that you have to deal with when you transition from the kind of laboratory environment of a scientific investigation into an actual system. And OK, so one thing is the real world factors in which the system is deployed. Another one is the scale. So as we said, this needs to be able to power the queries that people will fire off simultaneously from many different places around the world. And it has to all still be as seamless of an experience to be useful because sometimes you need immediate alterations to the route if anything important happens or anything changes.

12:22Yeah. So if we speak maybe the training phase and the inference phase, can you walk us through like from a training point of view? So you had this hypothesis, let's maybe model out roads using graphs. Talk us through what sort of data did you need, the scale, the training, just to build out a model and then actually deploy it and think about inference. Yeah, so I don't have all of the details off the top of my head, but what I can say is that we looked at, and you can see a lot more details in our research paper, which we published together with the deployed system. and there for the research study we focused specifically on I think four metropolitan areas like Los Angeles, San Francisco, Tokyo and I think the last one was Singapore but I'm not completely sure.

13:13So we chose different representative metropolitan areas that would correspond to different styles of traffic and so on where we would also expect a lot of queries to come from Right. So we kind of drive our base conclusions by careful, controlled studies in those regions. And then we try to see how this works when you train it on the entire system. The training itself isn't that big of a deal because, you know, as we said, roads get built and demolished. Sure. But the dynamics of traffic stay roughly consistent across, you know, relatively medium range timescale. So you don't have to retrain the model that often.

13:53Right. The inference is actually where things get interesting, right? Because as we said, this needs to have very stringent latency requirements. And this is actually one of the main things that trips machine learning scientists is that people actually don't care sometimes about a 2 % to 3 % improvement in accuracy if it means I cannot serve the system to that many users, right? So actually, the solution we came up with, and it's also briefly detailed in our paper, is most of the times when you invoke the system on a route where the graph neural network might be appropriate to use, most of the times you won't really trigger the graph neural network itself, but rather what the team does is it periodically will run the graph neural network on the most important road segments and cache the results, right?

14:38So then basically you can just very quickly get pre-computed results that are only ran periodically, not at every single query's time. Obviously you lose a little bit of accuracy that way, but you save a lot in terms of latency and the gains are obviously still there. Yeah, I love how caching remains a number one tool for a software engineer, isn't it? It's like, you gotta make use of it. So that's fascinating. And what's the impact of that work? So it's clearly deployed. If I use Google Maps now, it's basically using the fruits of that research? Yeah, basically, as far as I know, even today. It's been quite a few years since that research landed, but that system is still used as part of the travel time predictor.

15:26And the impact really is the reduction in negative outcomes, right? And here we can define a negative outcome as the estimate you got from maps was, say, more than 10 minutes off compared to how long it actually took you to travel there, either faster or slower, right? Both sides might be problematic. And the reason why this is such a big impact, it's okay, you think often on the level of a single user like yourself, if you want to travel from A to B, you might, you know, be able to not miss your meeting if you can better take into account how long it will take you to travel, even though it doesn't seem like there's a rush hour immediately.

16:01But I encourage you to think more broadly than that, because many companies rely in some part or in whole on the Google Maps API to give you travel estimates for services that you have acquired. So, for example, food delivery services, rideshare services, a lot of those rely on a system like Maps to tell you when your car is arriving, when your food is arriving. And obviously, having better estimates on those can be immensely helpful to improve customer satisfaction across the board, right? super so if you could summarize what are the sort of maybe key insights are could be useful for other data science and machine learning teams at other companies what would those be out of that that work um so i would say the main takeaway and i think we will touch upon this takeaway again in some other works that we'll talk about is that like people who do data science and machine learning were often really excited just by seeing one number being bigger than another number if that makes sense especially for people who were like you know classically academically trained like the two of us you know but like there is a gap between seeing one number being higher and actually building something that people want to use.

17:31And it might feel like a lot of the tediousness comes from that part, but it is actually arguably the most rewarding part. And the more of us realize that and actually work to bring things out of prototype and actively talk to the stakeholders and figure out what would it take for you to start using this, I feel like it would accelerate the useful adoption of AI systems a lot more. Because we have, I mean, just look at the AI literature in journals and conferences. There's so many papers with fantastic potential in amazing areas like, you know, biology, medicine. I mean, vehicle routing is also one of them.

18:11There's a lot of papers with really innovative ideas, but they're only tested on some small subsets of possibly relevant data. And you publish your paper and you kind of do what you need academically and you kind of stop there. I feel like there should be more of a drive, right, to go out of that prototype stage and into the actual real world. And that's where you pick up, I would say, some of the most useful skills that you wouldn't otherwise have. Yeah, that resonates. So focus on the user, the customer, non-functional requirements. Those are things that drive ultimately adoption satisfaction.

18:48So those are great insights. So you've worked on many projects. So let's talk about the next one, right? So you've been involved in AI for mathematics. Can you tell us a little bit about that research? The main idea of this work is that we wanted to create an artificial intelligence tool that mathematicians will be able to usefully leverage to do their research. And this is really important because, once again, there are many, many papers on the archive trying to apply AI on a mathematical problem, showing that they have better accuracy than some previous method and just leaving it at that. But a mathematician cannot publish a top research paper saying I've built a system that's 85 % aligned with this theorem.

19:34You either prove the theorem or you don't. There doesn't exist this middle ground in mathematical research. Like we really are looking for the hard truths with respect to a given set of axioms, right? So because of this, actually, when we started this project, I would say there was a really significant dose of skepticism among mathematicians. Like precisely because of this kind of culture we had in machine learning, they believed that it would be really hard for an AI system to meaningfully impact mathematical research. Fortunately for us, there were several really cool guys who were top tier mathematicians and who agreed to take a chance on us.

20:12And we started a collaboration with Mark Lackenby and Andra Shuhash from Oxford and Jordi Williamson from Sydney. And we worked on some really kind of diverse areas of mathematics of interest to them, including geometry, topology, representation theory. and the unifying kind of aspect of all of the applications we pursued is the fact that mathematicians often have to reason about really complicated structures, right? And sometimes, you know, even though we have the power of computers that allows us to like visualize a lot of these things and observe them, even though they're really complicated, it's still really hard to pick out from this really huge structure what is really important, like what drives the behavior of that structure, this mathematical object, so that you can truly understand it, like really try to see the forest for the trees, right?

21:12And specifically, the problem that I was most involved in with Jordi Williamson in representation theory, it concerns mapping a graph to a polynomial. Now, a graph in this particular case gets very complex very very quickly so even though we know how to build a graph and polynomial pairs very easily very quickly the graph gets super complicated and the polynomial barely barely gains an extra power but in order to prove a theorem relating these two you need to find the formula that works for all possible powers why was it an important problem Right. But yeah, our system is able to take this complicated graph and then predict the coefficients of the polynomial with reasonably high accuracy.

22:01Once again, not 100 % accuracy, so you can't use this model to prove the theorem directly. But you can then interrogate the model and ask it, what were the important parts of this graph that led you to make this decision? All of these systems are differentiable. They rely on gradients. So we can just measure which nodes and edges contributed the highest gradient. And we show those nodes and edges to the mathematician. As you can imagine, this was far from a clear picture. Like it was a very messy picture of nodes and edges. However, there were only 15 nodes now rather than, you know, a thousand.

22:34So we believed, okay, maybe if we show sufficiently many examples to Jordy, he would be able to pick apart some interesting things that are happening there. And actually that's what happened. So Geordi stared at these smaller graphs for several weeks and ultimately realized that inside all of them, a secret kind of hypercube-like structure was hiding. And when he discovered this structure, this gave him everything he needed. He went away, he proved all of the relevant theory and showed that under certain conditions, these hypercubes will be sufficient to settle the grand conjecture, right? We haven't completely finished proving this, but already what we have done made a huge dent in an area where humans haven't really been able to make progress for many many years that's super cool so in this story then if we talk broadly about ai for accelerating science of ai for accelerating innovation where do you you think we are on that journey right clearly google deep mind is doing a lot of work in this space you know Are we at the beginning?

23:39Should we expect like, this is going to be the new normal, you know, every PhD student like you and I 10 years ago, you know, we're going to be using AI to like accelerate discovery now. Like, how do you see it? Right. So I would say it largely will depend on what problems you're studying. Some of them will obviously be a lot more amenable to this than others. So even our initial math study, this is also the part of why it's so important to engage the domain experts. Like we talked with Mark, Andras and Jordi to tell us which problems from their areas that they're experts in are both like approachable enough that we could throw a deep learning system at them and expect meaningful returns.

24:19Like there was a lot of data, structures got complicated pretty fast, so on and so forth. While at the same time being a task of high enough interest to mathematics that we could publish it in a top tier journal if we were to make a meaningful dent. Right. So I think the communication with the domain experts was always paramount to make sense of it. Where does it make the most sense to do? But, yes, I would say we are definitely now in a phase where mathematicians are no longer skeptical of AI accelerating their work. So we've done that initial push. And now I think it's a lot more direct of an idea to be talking about.

24:53And, you know, we find ourselves, so I can speak from personal experience, we have actually leveraged Gemini as basically a theorem proving assistant of sorts, where in one of our recent papers, we have leveraged it not only to help us discover some interesting structures, but also to help us prove interesting things about the structures. And the Gemini system did not like set up the proof and do it all by itself. We still gave it guidance in terms of like what kinds of outcomes we're after, what we believe the solution is, help us come up with some proof sketches and stuff. But yeah, it's basically at a point where if you have the know-how to steer it meaningfully, it can totally accelerate that part of the mechanical part of the work of just figuring out arguments to apply and prove stuff, which would take a lot more time to someone who's not like doing mathematics on the daily like myself.

25:46And I should stress, this is a paper that we have published recently at ICML, which is one of the top machine learning conferences. So certainly already the top tier events in various branches of science, not only AI or mathematics, are leveraging the current generation of AI systems to meaningfully assist in the process of scientific discovery and contributing to the body of knowledge. Are we at a point where you can have a full-fledged AI scientist? we are not. And even when we do, I think the direction from humans is going to be really important because, you know, you can have even today a theorem prover that will prove you a thousand theorems without any problems.

26:26But like, how do you decide what's important? How do you decide what are the problems that we collectively care about the most? I feel like that direction is always going to be super, super important. But already today, even though the systems are not perfect, you can totally use them if you guide them the right way. Great, great. You You mentioned an AI scientist. How would you define what an AI scientist is? So it would basically be a system that's capable to a larger or smaller extent of closing the loop, if that makes sense. So like starting maybe with some initial direction about an area or a topic that you'd like to explore.

27:04They could start by doing a bit of a literature review to figure out what are the gaps in our understanding, proposing some targeted experiments, maybe even executing those experiments themselves. in the loop. And then based on the outcomes of those experiments, you know, proposing maybe some theory or follow-ups or maybe even writing papers on their own, you know, any part of the scientific discovery pipeline that you could imagine having an AI system helping out with, that would fall under the definition of an AI scientist for me. Of course, for many people, the true AI scientist is a system that goes end to end.

27:43So like from the initial prompt, it goes away, does everything and like returns with a paper or something like this. Yeah. Great. It's fascinating to hear this story because AI is clearly for everyone, regardless of, you know, whether you're just using Gemini, you know, for your emails or for navigating on Google Maps all the way to being a senior researcher and you're using AI yourself to not only speed up your research, but kind of come up with new angles that you can pursue. So I think that's a really good insight, AI is for everyone. Amazing. Where do you think it's going? Fast forward a couple of years.

28:28I would say kind of from the trends that I'm observing, more and more people are being mindful of these kinds of limitations I've pointed out, particularly because there's a lot more efforts trying to build AI scientists in one form or another, that, you know, whenever there is a critical mass of people appreciating that a problem is important, we are probably going to be in for a couple of breakthroughs to help us kind of make progress on those. Will it take two years? Will it take more than two years? I cannot really speculate. Research sometimes moves in unexpected speeds, both higher and lower.