The AI-Robotics Revolution: Robots That Think Like LLMs

0:00Today on the AI Breakdown, we're discussing some major advances in AI for health and medicine. Before that on The Brief, a big discussion of the convergence of robotics and LLMs. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our Discord, and our newsletter. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. Today we kick off with the convergence of, on the one hand, robotics, and on the other hand, artificial intelligence and more specifically, large language models.

0:36Now, one company at the very center of both of these big shifts is Google's DeepMind. At the end of last week, they introduced something they call RT2. Robotic Transformer 2, or RT2, is they say a novel vision language action or VLA model that learns from both web and robotics data and translates this knowledge into generalized instructions for robotic control. The TLDR here, as simply as it can be put, frankly oversimplifying it, is that rather than training robots on first-hand data across every object and environment and situation that they might encounter, RT2 instead translates all sorts of web and robotics data into more generalized instructions that robots can use to make sense of the world around them.

1:17And so just like generalized LLMs can do more than just a specific set of tasks that they were trained for, RT2 is also increasing those generalized capabilities. DeepMind sums up, RT-2 shows improved generalization capabilities and semantic and visual understanding beyond the robotic data it was exposed to. This includes interpreting new commands and responding to user commands by performing rudimentary reasoning, such as reasoning about object categories or high-level descriptions. We also show that incorporating chain-of-thought reasoning allows RT-2 to perform multi-stage semantic reasoning, like deciding which object could be used as an improvised hammer, a rock, or which type of drink is best for a tired person, an energy drink.

1:54So what are the types of skills that come with RT2? Well, first, by way of understanding what Google DeepMind actually did, they say they performed a series of qualitative and quantitative experiments on over 6 ,000 robotic trials. The three categories of skills that they defined were symbol understanding, reasoning, and human recognition. Each task, they say, required understanding a visual semantic concept, as well as the ability to perform a robotic control to operate upon that concept. So for example, pick up the bag about to fall off the table. the robot has to understand what bag is about to fall off the table and then respond to the command to pick it up.

2:29Google reports that across all three categories, they observed increased generalization performance and more than a 3x improvement compared to previous baselines. Now, the New York Times wrote about this as well, which is why so many people have been chattering about it. The New York Times piece is called Aided by AI Language Models, Google's Robots Are Getting Smart. Giving a storyteller's flair to the demonstration, they write, A one-armed robot stood in front of a table. On the table sat three plastic figurines. a lion, a whale, and a dinosaur. An engineer gave the robot an instruction, pick up the extinct animal.

2:59The robot whirred for a moment, then its arm extended and its claw opened and descended. It grabbed the dinosaur. As the New York Times points out, until very recently, this demonstration would have been impossible. Robots weren't able to reliably manipulate objects they had never seen before, and they certainly weren't capable of making the logical leap from extinct animal to plastic dinosaur. However, they say a quiet revolution is underway in robotics. As the Times reports, right now Google isn't planning on selling these RT2-powered robots or releasing the model more widely. However, they do believe that eventually this type of robot that has a built-in language model is going to find its way into many, many use cases.

3:34Now, just to leave an anonymous note, the New York Times concludes their piece, if you're the kind of person who worries about AI going rogue, making robots that can reason, plan, and improvise on the fly probably strikes you as a terrible idea. But at Google, it's the kind of idea researchers are celebrating. After Gears in the Wilderness, hardware robots are back, and they have their chatbot brains to thank. Next up, a story that is at once nerdily cool, as well as possibly enraging for the anti-tech set. Gizmodo is reporting that Dungeons & Dragons owner Hasbro has teased the idea of integrating powerful AI-driven game mechanics for D &D.

4:05This comes as part of a press release for a partnership announcement between Hasbro and Explored. That press release says that the partnership would allow Hasbro to deliver innovative gameplay to our players and fans, limitless digital expansions to physical games, seamless onboarding, and powerful AI-driven game mechanics. When asked by GamesRadar for more on what AI-driven game mechanics meant, a VP at the company said that AI might be used to do things like, quote, generate experiences that could react to player decisions right away, and potentially streamline rules to make it easier on newer players.

4:34Gamers taking to Twitter were unsurprisingly upset. Alizair Stewart says, almost the first of the month, Joe Hasbro. Shall we unleash the stupidity kraken again? Jack Murphy says, Kind of sad watching this game become a victim of its own popularity, as Hasbro makes every mistake possible as it tries to squeeze blood from a stone. H Hooligan says, Let's be honest, technology has made character creation and gameplay easier, but this is nothing more than Hasbro and WotC trying not to pay humans if possible. And Cora Buellard simply says, Stop with this AI crap no one wants or needs. I don't know, man.

5:05I'm kind of excited. Let's see what they do. Moving on to our next topic. It seems like every weekend in the Bay Area, there is another big hackathon, often taking place at the AGI house, which builds itself as the machine learning hacker house, bringing hackathon life back to Silicon Valley. Jeremiah Awang was at last weekend's Anthropic hackathon, and the list of projects are pretty interesting. There was a real-time fact-checking platform for live broadcasts, Dr. Claude, which is effectively a personal AI physician that uses your own data, ImmigrantFirst.ai, which helps immigrants figure out what information they need based on the government of whatever jurisdiction they're in and what they have to file, and the winning project, which was called Claude Scholar.

5:42Jeremiah summed it up, finds insights in existing data from molecular scientists, upload data, PDFs, which you can then summarize information, apply for grants, and generate new info such as molecule variants. Finally today, we continued to see policy efforts in the United States around AI. Last week saw the introduction of the CREATE-AI Act, which stands for Creating Resources for Every American to Experiment with Artificial Intelligence Act. The CREATE AI Act would establish the NAIRR, the National Artificial Intelligence Research Resource. And if you're exhausted with these acronyms, basically what this is trying to do is to ensure that it's not just big tech companies that can actually do cutting-edge AI research.

6:19The NAIRR is designed to be a, quote, shared national research infrastructure that provides AI researchers and students from diverse backgrounds with greater access to the complex resources, data, and tools needed to develop safe and trustworthy artificial intelligence. Sponsoring Congresswoman Anna Eshoo said, AI offers incredible possibilities for our country, but access to the high-powered computational tools needed to conduct AI research is limited to only a few large technology companies. By establishing the National Artificial Intelligence Research Resource, my bipartisan Create AI Act provides researchers from universities, nonprofits, and government with the powerful tools necessary to develop cutting-edge AI systems that are safe, ethical, transparent, and inclusive.

6:56The four primary goals are to, one, spur innovation and advance the development of safe, reliable, and trustworthy AI research and development. 2. Improve access to AI resources for researchers and students, including groups typically underrepresented in STEM 3. Improve capacity for AI research in the United States 4. Support the testing, benchmarking, and evaluation of AI systems developed and deployed in the United States Now, ultimately, what this represents, I believe, is the fact that AI isn't going to be just one big set of banner legislation Sure, we might get comprehensive legislation from the likes of Chuck Schumer, who has been working on exactly that But I think it's far more likely that along the way, we see lots and lots of these very specific smaller bills, which are highly focused, have a good chance to pass, and address some aspect of the AI ecosystem that politicians want to get addressed, perhaps on its own terms, without having to argue in the context of a larger, more comprehensive, and thus more controversial bill.

7:46In this case, the concern is that the high cost and barriers to entry of developing AI and of advanced AI research could reinforce inequalities that exist already, thus an attempt to expand access and have this next technology wave not necessarily reflect the biases of the last. Anyways, guys, that is going to do it for today's AI Breakdown Brief. Thanks as always for listening or watching, and I'll be back soon with the main AI breakdown. Before we get to the main episode, I want to tell you about today's sponsor, NetSuite. I know many of you guys are entrepreneurs, executives, managers, business leaders who are trying to figure out how technology is changing the world and how it can change your business.

8:24Given that, I am thrilled to have NetSuite as a sponsor of the AI Breakdown. NetSuite gives you the visibility and control you need to make better decisions faster. It is the software superpower behind so many of the world's most successful companies. And for the first time in NetSuite's 25 years as the number one cloud financial system, you can defer payments of a full NetSuite implementation for six months. That's no payment and no interest for six months, and you can take advantage of this special financing offer today. Listen, NetSuite is number one because they give your business everything you need in real time, all in one place to reduce manual processes, boost efficiency, build forecasts, and increase productivity across every department.

9:04I know that if you are listening to the AI breakdown, you understand intuitively and deeply just how much data matters to any modern business. Having all of your information in one place can be the difference between making the right decision and making the wrong one. I think it's awesome that NetSuite has this new offer designed to really make their suite of tools available for all the businesses that need it. So, if you have been sizing up NetSuite to make the switch, then you know that this deal is unprecedented. No interest, no payments. Take advantage of this special financing offer at netsuite.com slash breakdown.

9:36Go to netsuite.com slash breakdown to get visibility and control you need to weather any storm. That's netsuite.com slash breakdown. And with that, let's get to the show. Welcome back to the AI Breakdown. Today we are talking about AI advances in health and medicine, including a new multimodal model from Google, as well as research inspired by Jurassic Park. Cesar de la Fuente is a presidential assistant professor at the University of Pennsylvania. Recently, he tweeted, My lifelong dream has been to use machines to accelerate discoveries in biology. Using AI, we discovered antibiotics in Neanderthals and Denisovans, our closest hominid relatives.

10:15Molecular de-extinction opens new avenues for drug discovery. So I first saw this reported in Nature, and the piece was called AI Search of Neanderthal Proteins Resurrects Extinct Antibiotics. So this is obviously, as someone who's interested in both AI and deeply in history, pretty much the most clickbaity, catnip type of title you could imagine. However, the research is useful for much more than just a good article title. Nature.com gives the context really simply. They write, Antibiotic development has slowed over the past few decades, and most of the antibiotics prescribed today have been on the market for more than 30 years.

10:49Meanwhile, antibiotic-resistant bacteria are on the rise, so a new wave of treatments will soon be needed. I wasn't joking when I said this was inspired by Jurassic Park. De La Fuente said, We started actually thinking about Jurassic Park. Why not bring back molecules? Now, the specific molecules they were interested in were peptides. Peptides are a short protein subunit and often have antimicrobial properties. So what De La Fuente and his researchers did was train an AI model first to recognize human protein peptide sites, and then second, use that model against publicly available protein sequences from our ancient relatives, including the Neanderthals and the Denisovans.

11:26From there, they used the properties of previously known antimicrobial peptides to predict which of the new peptides from these ancient human relatives might help kill bacteria. Researchers then tested dozens of the peptides that AI had suggested to see if they could kill bacteria first in laboratory dishes, but then with six of the highest potential examples, tested them on mice that had been infected with a bacteria that is a common cause of hospital-borne infections among people. The results were mixed. The peptides halted the growth of the bacteria, but none actually killed it. And while the results might not have been a slam dunk, the research team think that one, using these high-promise peptides as a jumping-off point could reduce the time to drug discovery, and second, they think that the AI model they used might be able to be improved to actually have a higher degree of predictive success.

12:11Now, if we started in our ancient past, we now zoom forward to our near future. On Sunday, Lior at AlphaSignalAI tweeted, Incredible news! The first generalist medical AI system is out. DeepMind just announced MedPOM-M, a multimodal generative AI model that understands 1. clinical language, 2. imaging, 3. genomics. What he's referring to is a new research paper from Google that was called Towards Generalist Biomedical AI. The abstract of the paper reads, medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery.

12:55To enable the development of those models, they say, they introduced a new biomedical benchmark, which they called MultimedBench. MultimedBench encompasses 14 different tasks that range from medical question answering to radiology report generation and summarization, and then they introduce something they call MedPOM multimodal or MedPOM-M. So basically what this research team is attempting to do is in the same way that we are now deploying AI models that are generalist, i.e. we can use them for a variety of tasks, not just a very specific task they were trained for, this Google team is trying to do the same thing in the field of medicine.

13:28TLDR, the researchers had some really promising results. The model showed zero-shot generalization on novel medical concepts and tasks, and even emergent zero-shot medical reasoning. Now, zero-shot refers to a type of machine learning where a model is able to correctly make predictions or decisions on data it has not been explicitly trained on. The term zero-shot comes from the fact that the model receives zero examples or shots of these specific tasks during training. This is important in the context of this generalist model, as it demonstrates a model's ability to generalize knowledge from known tasks to unknown tasks.

14:01Now, one of the big examples they used of the performance of this model in their tests was a radiologist's evaluation of model-generated chest x-ray reports as compared to reports produced by radiologists. In a side-by-side ranking on 246 retrospective chest x-rays, clinicians preferred MedPalm-M reports over those produced by radiologists in up to around 40.5 % of cases. Now, obviously there's a long way to go, but having two-fifths prefer this generalized models version rather than a specialist created version by humans suggests that there's a lot more promise here. Now, this relationship between AI and medicine is one that is just growing every single day.

14:37In July, Dr. Daniela Lamas wrote a guest essay for the New York Times called There's One Hard Question My Fellow Doctors and I Will Need to Answer Soon. In it, she says, the idea of a computer diagnostician has long been compelling. Doctors have tried to make machines that can think like a doctor and diagnose patients for decades, like a Dr. Howe style program that can take in a set of disparate symptoms and suggest a unifying diagnosis. But early models were time-consuming to employ and ultimately not particularly useful in practice. However, she points, generative AI systems are different. They are not, quote, the same as looking up a set of symptoms on Google.

15:11Instead, these programs have the ability to synthesize data and think much like an expert. To date, she says, we have not integrated generative AI into our work in the intensive care unit, but it seems clear that we inevitably will. Interestingly, Dr. Lamas also points to x-rays, the subject of that MedPalm-M test, as a likely example where AI might come to the fore sooner rather than later. She writes, one of the easiest ways to imagine using AI is when it comes to work that requires pattern recognition, such as reading x-rays. Even the best doctor may be less adept than a machine when it comes to recognizing complex patterns without bias.

15:42Echoing the concerns of people in other fields where they worry that the rise of AI will undermine fundamental skills, Dr. Lamas says, there is the real possibility that doctors in training could lean on these programs to do the hard work of generating a diagnosis rather than learn to do it themselves. If you've never sorted through the mess of seemingly unrelated symptoms to arrive at a potential diagnosis, but instead relied on a computer, how do you learn the thought processes required for excellence as a doctor? Ultimately, Dr. Lamas doesn't have the answer, but in the context of an ever-advancing set of medical AIs, it's one that's going to come up more and more.

16:14That is going to do it for today's AI Breakdown. If you're enjoying it, do me a favor and text one person a link to this show or another recent show that you think they would like. The AI Breakdown community is one of thoughtful, intentional people, and I'd love for you to invite someone in. Thanks for listening as always, and until next time, peace.