#335 Sriram Raghavan: Why IBM Is Betting Everything on Small AI Models

29:27It's a very simple check to do. If it doesn't show up, go back to the model, try again. These things can be done. You will see now order of magnitude reduction in model performance. So we certainly see these two vectors continuing to push the needle on how small models are going to do. Today, because of the way the field has evolved, prompts have become sort of this only lingua franca. But we're at a point where we're building more and more sophisticated applications. If you're only building a Q &A application, prompt is okay. If you're building a full-blown agent, talking to tools, making planning calls, iterating, reifying, I think you're going to see programming abstractions.

30:06And we're very actively working on some of those programming abstractions. And that's done by the developer in the code of the application. At the application. But we are going to give them now programming frameworks. So we have started to use the word, by the way, generative computing. So we keep talking generative AI. We really believe we are on this cusp, and you will hear more about this, I think, when you're there. You're going to see us talk about generative computing, because I think we are at the shift where we went from, hey this cool self supervised model LLM cool what can it do then we went from that to okay you know what a model is part of an entire platform so that's where you started having platforms from us and others we built Watson X where models are important but you need all this plumbing around the models we talked a lot about models as a data representation we said it's important to get enterprise content into those models that's why techniques like InstraClab you use rag to give the model enterprise that's not appearing I think we are at a point where we are now, all of those are true.

31:09It's a powerful self-supervised representation. It's core to an AI platform. It is a data representation. But I think we are at the point of saying a model is now a new type of computing element. If it's a new type of computing element, a computing element has abstractions. It has a runtime. It has ways to speak to it. And I think that is what we see starting to emerge. I think people are very quickly realizing. So we think that just as we talk about classical computing and quantum computing, we are very comfortable now starting to use the word generative computing. Now only difference is that unlike quantum, it's not an entirely new hardware off on the side.

31:50It's a hardware we understand. But I think treating the model as a computing element now allows you to ask, how do I program this computing element? A programming a compute element doesn't have to mean writing five pages of English. Programming a computing element can mean here are the abstractions. So that's where we are pulling and it's motivated by what we just discussed. If I can do that I can do very very well with much much smaller models and that drives efficiency. Yeah and when you say generative computing is there a model that's that's generating the program I mean actual code to do these things?

32:30No so we are using generated computing to what we described which is a program that you use to work with the model but not as if you just have to give it a prompt but you can actually use instruction so as a concrete example let me let me give you a flavor of water generative computing program could look like so we released along actually with granite 3.3 instead of what we call LoRa adapters right these do one thing very very well so they you know there was a LoRa adapter that does hallucination detection. There's a LoRa adapter that can rewrite queries better for better search performance. There is an adapter that does uncertainty quantification, which says these are all trained adapters for those specific things, right?

33:14So you can think of these LoRa adapters almost like plugins stuck on top of Granite 3.3. So you take Granite 3.3, you can have three, four, five adapters enabled. Now a generative computing program will actually, where a program you say what I'm building a rag application and I want to turn on or off one or more of these LoRa adapters. So that's what we mean by generator point. Now why is this interesting? This is interesting because I could do this whole thing without these individual LoRa adapters with a big model and in English say I want you to do hallucination check. But now that big model has to be trained to get all of this done.

33:52It has to rewrite the query, it has to do hallucination detection, it has to do uncertainty quantification. But now by decomposing it into a good base model and individual LoRa adapters, the programmer controls. They stitch a workflow together, turn it on and off when needed. I'm working in an environment where uncertainty score is not important, I don't need that LoRa adapter. So that's what we mean by a generative computing program. So the model is really wrapped by a runtime. So you no longer are talking raw to the model. You're talking to a runtime that wraps the model. The runtime exposes abstraction that lets you on and off functionality.

34:31And then the runtime enables or disables these lower adapters. Eventually you're running an inference through. Finally that's the operation. But you are setting up that inference based on these flags and other controls. Right and turning on and off these elements, turning on and off inference time compute for example. Is that automatic within the model does it decide? The runtime. Now the runtime associated with the model does it. Right. It's not the developer in advance. No. Yeah. No. So you just set up that environment and the developer comes in and uses this abstractions. So for example take this one.

35:13You have a requirement to you have a requirement that your output looks a certain way. The developer can say here is a little function that's going to check that you meet the requirement right. It can be a little piece of a program. So think about it if you didn't do it this way then you have to teach that model how to do this requirement. That's why you end up writing these laborious English using examples that show this is the right this is wrong because that's the way for the model to learn. On the other hand all of that is useful if the model is directly exposed to an end consumer in a consumer use case.

35:47We're talking about enterprise applications that developers are writing. So if there is a simple check that says, this document's output must always end with date and IBM confidential at the end. That's a simple check that any programmer in high school can write. You declare a little function. Okay, you write your generative computing program. You can iterate through, and you will achieve the same effect. So that's what we mean by the generative computing program. What's interesting is there are now more and more efficient ways to do this. So we talked about LoRa adapters. One of the challenges with LoRa adapters is, yes, they allow you to build these modular pieces of functionality.

36:29But anytime you invoke a LoRa adapter, it's like you're processing a set of tokens. Then you invoke a LoRa adapter. While processing the set of tokens, you would have generated a KV cache for the base model. Soon as you invoke the LoRa adapter, you'll have to throw out that KV cache because you'll have to redo the KV cache because the LoRa adapter has been trained on its own. So you'll have to repopulate that. So you've been developing an interesting technology called activated LoRa. And the goal of the activated LoRa is, can I share KV cache with the base model? So you can turn me on and I can actually reuse the work that was done by the base model in processing some amount of tokens.

37:10then just do my thing that I need to do, move on and a base model can continue to move on. So this is now squeezing out even more. We have seen activated LoRa's can give 2-3x advantages between dedicated LoRa's. So you see where I am going with all of these things. They are all about finding powerful ways to bring down. So we are very, how to say, very convinced that abstractions around models are going to emerge. We're experimenting with a set of abstractions, but these are going to emerge and they are going to be supported by both the programming model and then these kinds of innovations. LoRa's, activated LoRa's, I'm sure there are other innovations waiting that will allow us to write really complicated things.

37:57And if you step back and look at what's happening, things like what people call agents. Already agents contain some form of classical procedural imperative code. They always do. Sure. Right? And what people are figuring out is what should I do and what can the model figure out on its own. But that's a little bit of a contract on what I should do. Certain things today we are asking the model to do. We may be end or off just doing it outside because then you can still achieve a lot of the same functionality but get it done with a small model. So I think that's the journey we are on and that's what's exciting to us.

38:29That's why we think we are at a point where it's no longer we're not just doing AI. We actually have a new kind of computing element that we're going to know how to interact. And actually, that leads me to a question I wasn't going to get into to the end, but IBM's commitment to open sourcing these models. Meta has just asked, you know, people to help raise money for training future LLAMA models. And that's something I've kind of been used to ask Yan La Koon about. I mean, how long are they going to stay committed? this is such an expensive enterprise if you're just giving the models away. If you're doing it with capable but smaller models, you can do many more than Medican with LAM.

39:21Is that part of the strategy with these small models? Yeah, I think, look, our strategy was about small focused models from the get-go. And I think that was motivated less necessarily by whether that's the only sustainable way to put it in the open or not. We think that's the only sustainable way to get enterprises to at scale adopt AI. Because finally, they have to run these models. Yes, I could spend the capital and train the biggest model, but if that isn't necessarily that going to get deployed, and particularly with IBM strategy, we are looking for it to run across a hybrid environment. So we wanted to work on multiple clouds, on-prem infrastructure, diverse set of hardware.

40:00We've been very public about that too. run on NVIDIA, run on AMD hardware. We have our Spire chip where we run on our own mainframe. So hardware agnostic multiple environments. It's very important and that means that being able to optimize the model to get it running very very efficiently is really important. So if I deal with a huge object that isn't easy to manipulate, I'm basically shooting myself in the foot with kind of a strategy. So we were on small models anyway. So I don't think it was necessarily one way or informed by this and you can see that even in the philosophy I have taken, Granite 3.3 for example included a speech model, a vision model, a 2 billion better document processing model.

40:42Because eventually what is going to happen is you are going to solve a problem. To solve a problem or build an application you are going to have AI models work together. Some cases they are orchestrated by an agent, other cases you are stitching together. I have a document processing pipeline, I use a 2 billion parameter model. On the run time I use an 8 billion parameter model. If I have speech input, I use a speech model. I don't need one model to do all of these things. I can run these models differently. If that speech model needs to only run in a certain environment, I only need GPUs to run the speech model in that environment.

41:15If my document processing needs, I'm a hybrid cloud company, I'm using Watson X on Azure to run my document processing, that model needs to run on Azure. My application could be running on the mainframe, that model needs to run on Spire chip on the IBM mainframe. So this unique philosophy of fit for purpose decomposition we think is the right approach to build AI. So we've been on that path. Yes it does make it therefore much easier for us to iterate, rapidly release. Look at Granite 3. We released Granite 3 in October. 3.1 end of the year. 3.2 in January. 3.3 in February. We're going to be releasing Granite 4 by 2Q.

41:56That pace is possible because we have made this choice. And you can compare that to people who have made a different choice, whose release cycles tend to be six, nine months a year. It's just the way things are. Yeah. Yeah, I wanted to switch to more forward-looking, and not that this isn't forward-looking, but continuous learning in the search for... I mean right now although you've managed with these small models to train them at inference time, that doesn't adjust the weights in the underlying model. So there's no learning in the underlying model during inference. Correct. Inference scaling is on the top.

42:46Correct. the model remains the same. The only change is the LoRa. So you can drop in and take out LoRa. So that does give you runtime adaptability. Right, I see. Are you pursuing research? And I know that it's an active area of how you could update the weights of the underlying model during inference time to create a continuous learning loop. Right, so I think maybe to pass that a little bit. So I think first is the this notion of LoRa's and activated LoRa's already give us a way to to change notionally the weights of this composite thing instead of calling it a model. I want to call this as composite thing which consists of a base model and multiple adaptive LoRa's right.

43:36So already I get more flexibility than just having the one base model. I can turn it on and off. I can add a LoRa, I can remove a LoRa out of the system. So that already gives me modularity. So that's number one. Which means that I can deploy a system with five LoRa's. I'm collecting telemetry data of usage of the model. I can use that to go off and train another LoRa, plug it back into this model. So I don't necessarily have, it doesn't necessarily mean that every online learning I need to do has to go back and touch the base model. Now there are certain environments in which maybe I need to do that.

44:14So for example, I'm trying to collect chain of thought data or backtracking logic in an entirely new domain. I may need to seed some of that early on in the training of the base model, maybe in mid-training or something like that. So I may deploy a system, instrument it to understand how humans are doing that, collect the data, curate it, generate synthetic data and go back to it. That I think is a longer loop. But there are certain scenarios where I may be able to connect trajectory information like this. Just train a LoRa adapter that is trained on that and simply helps the model do that backtracking.

44:55Or I may be able to adapt a reward model that sits outside the base model that's trained on, Oh I see this is how humans backtrack in this domain. Let me get a reward model that's going to now teach the main model how to backtrack. And those reward models, because they are separate, I can update them and refine them on their own. So I think the way we are going to do online learning, until we figure out maybe a methodological innovation, is essentially by making the runtime that is actually serving AI highly modular. So reward models, LoRa's, I see as the mechanism to do that. Because every time you have to fall back on updating the base model, you're right, that's a long loop.

45:35Are there, have there been ideas around in real time updating the watch to the base model? Yes, but I haven't seen a practical deployable answer. Right now the base model is still a retraining activity. Right. And their small is easier to update, that's about it. I haven't seen anything fundamentally moving the needle. But I do think modularity gives you a good way to address that. Yeah, although what I'm talking about with continuous learning is that the model would increase its knowledge base without getting larger, maybe would have to get larger, I don't know. And so once you're working with it on inference, it gets smarter.

46:26I mean, this is, you know... The original, yeah, learn smarter. The hockey stick and intelligence, yeah. Correct. But I think what we are discussing is the mechanism for that to happen, right? So the long loop of that is let the system run, collect the data, then you come back and retrain the base model. That's the long loop. It's not automatic. It's the longest loop. The second faster loop is I can update some of these components quicker. And so I learn faster. I noticed that the model is not doing well for certain classes of queries. quickly collect the data, build an adapter that addresses that, build a better reward model.

47:01That's the second way to do that. I think there is a third way to do, which is at an application level. You deploy a model, look at where it's not doing well, and then use that data to train a second model and route. So I think at systems level, you can make the system more and more adaptive. I see. So I think those techniques are going to be very, very practical, because they allow you to react. Now, I do think that what you're asking for, which is this vision that the model discovers what it is not good at on its own versus a human being looking at it and deciding to train. I think the big piece there is this whole area of uncertainty quantification.

47:44So that's why we are working a lot on that. Because the first step to doing that is for the model to have a principled way to know what I'm good at, what I'm not good at. And if there is a region of space in this knowledge that I'm consistently having low uncertainty, then you can figure out what action to take. Today, all the actions are human driven. The human decides when to come back and retain Allura. You're talking about a system that automatically recognizes its faults and goes there. I still think that is still in the realm of innovation and research right now. But possible. But possible.

48:17But certainly possible. I view uncertainty and quantification as a significant element. Because that's the first step. You've got to know what you're not good at. If you continue to hallucinate your way and you don't know, then you don't know you have a problem. So uncertainty quantification, because remember that if the model produces, if a system produces an output, and the model does not get a judgment, it doesn't even know whether it did well or not. Now today, some other external feedback signal says, I'm getting customer complaints. I'm not resolving these tickets, whatever they are. And a human looks at it and comes back and fixes it.

48:51So for the model to learn, the model has to automatically know what it is not good at. And that's why I think uncertainty of quantification is actually a really, really important piece of the puzzle for self-improvement. Okay.

49:07Yeah, I'm just... Yeah, this was a question off of that. if inference time weight adjustment is possible, would it happen at layer-specific updates? How fine-grained would that weight adjustment be? Or would it be, I mean, the problem with right now with catastrophic forgetting is it may update overwrite previous knowledge. Right. But would it be done at a very granular level, like different nodes or different layers? I think the reason I keep going back to LoRa is that's actually a principle way to do that. Because if you draw the box not around the base model, But around the base model with the LoRa's, you are adjusting weights.

50:07But you're adjusting weights by moving from a certain computation into activating a LoRa, which does something different. It's sort of like inserting something at that point. So that's really what you're doing. So at a system level, LoRa's do give you an opportunity to update the effective network that is giving you the answer. That's different from... And the reason I think that is useful is, As you said, changing the weights of a base model is complicated because you took something that was created by a massive optimization, which is balancing many, many things. You're making some local changes whose implications you don't know.

50:49That's what leads to all sorts of challenges. The philosophy with lower eyes, let's decompose it to specific pieces of functionality. I want to teach the model a piece of knowledge. I put a knowledge lower eye. I want the model to be good at hallucination. So when you do it that way, you are isolating these things. And so you have less worry about conflicting with what was taught to the base model. You have individually optimized those LoRa's to do what they've done. And you're putting this together almost like a workflow. I think that's a more viable way to do what you're thinking about online learning.

51:24Because now it allows you to reason about this in a meaningful way. It allows you to separate functionality. So you know that if performance comes down when a LoRa was enabled, then you can go debug that LoRa. If I did this update of the base model, I don't even know how I would, let's say I did that, Debugability, traceability, all that becomes extremely high. So I keep going back to, I think the way we are going to get to online changes is really by not treating the AI system as one big network, But it's really a base network with lots of plugins. And we are probably early in the innovation associated with it.

52:07And so I think that's a more viable way in which we will get online learning. Otherwise, it becomes impossible to reason about it. These optimizations, because training is effectively a big optimization problem. Over all the data that you send, minimize this loss function. So it's very hard to reason about. Loras have the advantage that I optimize them for one thing and I plug it in. I trained another Laura, I optimize it for something else. I plug it in. It's much easier for me to reason about it, swap it in and out. So I feel like that, again this is just me speaking intuitively. Maybe there is a brainwave, a way to do this that I'm not thinking about.

52:44But I view this as a very pragmatic way to get the effect that you're thinking about. Yeah. Is the Granite series, is that the main thrust on generative AI right now? At IBM Research, it's bigger. So maybe I'll step back to talk about our overall focus, right? So one is certainly obviously the models themselves and everything we have talked about. A second big area for focus for us is on the data side. Yep. From an enterprise context, unstructured data is becoming important now because of Gen AI. So a huge focus for us is making our WatsonX.Data lake house really best of breed for unstructured data, which means thinking about all aspects of data management from ingestion to processing to ETL pipelines to security, access control, governance.

53:34So that's a big, big focus here with a lot of innovation from research. So you'll see, again, announcements around agentic rag, announcements around unstructured data processing pipelines and document understanding. That's driven by research innovation. That's a big thrust, which is how do you make enterprise data management ready for unstructured data. There's a second side to the data side, which is there's a lot of work in enterprises that data engineers or data stewards or people. And we call this broadly this term data ops, right? Yeah. Running, this is even with structured data. Forget unstructured data.

54:08Today, even before Gen AI, there's a lot of time spent in creating pipelines of unstructured data, cleaning them, morphing them, transforming them, moving it from one place to the other. So data ops is a very expensive piece. So one of the areas we're working on is how does agentic AI help those personas who are data stewards and data engineers do their job better, faster, so that's applying AI to data management. So on the data side, we sort of look at it as two right? Preparing data for Gen AI and then using Gen AI to make data management more efficient and effective. So that's the second thrust.

54:49The third thrust for us is everything to do with agents and agent middleware. So we believe where the world is going very much are just like our philosophy around models, right? Yeah. We are about fit for purpose models that you can pull together. Same for agents. We think the way the world is going to be built is multiple agents you have to orchestrate to get your work done. Because it's much easier to securely build an effective agent that does one thing as opposed to one humongous agent that does everything under the planet. Which means managing agents, the lifecycle of agents, governing agents, the security of agents, orchestration of agents, that's going to become an important focus.

55:29So we have from our product portfolio Watson X Orchestrate is our flagship product for agentic middleware. And a lot of what we doing from research is looking at problems like observability of agents. Because you have to look at agents in action just as you observe IT systems. How do you do agent ops and observability? How do you securely make tools available to agents? How do you look at security and authorization of agents? It's a big issue because today if you're going to have agents. So imagine Craig you're an IBM employee. I put in front of you an agent that lets you do HR tasks. Now that HR agent is going to be equipped with tools to talk to HR systems, make operations changes.

56:14Those tools have to only have your authorization. So that you are only allowed to do. So securely delegating authorizations and permissions is a big area that touches on the overall security fabric. So that's a big focus for a security governance and operational lifecycle of agents and agent orchestration. is an area. And we are building also agents in three specific domains. Those are where we have IBM businesses. So we have a big focus on agents for software development, but not general purpose software development. We have doubled down on areas where we have a big IBM business. We have a huge franchise on COBOL on the mainframe.

56:52So we have Watson Code Assistant for COBOL, Watson Code Assistant for Java. So that building agents for software development in those domains. A second big focus for us is building agents for IT automation. We've had a rich portfolio now of IT automation, combination of organic and inorganic acquisitions, right? So across our Aptio, Instana, Turbonomic portfolio, we address the entire lifecycle of IT, from cost management to operations to security and compliance. So we're building agents to help those personas, whether it's a CISO, whether it's an SRE, whether it's the IT analyst managing cost, agents to help their work do better.

57:32And then the third area we are building agents is for our Maximo portfolio, which is used to manage physical assets. So Maximo today is used to manage physical assets like in factories and utility companies. And so there, the equivalent of the SREs, there is equivalent of maintenance personnel who have to keep these systems up and running, look at reliability, manage faults, building agents. So those three domains, software engineering, IT automation, and physical asset management. We're building agents. And as we build those agents, we understand what agent middleware is needed. So that drives our innovation in middleware.

58:07So models, data, agents, and three domain-specific agents. That's basically the... I should probably leave it there, but this has been fascinating for me. Thank you. It's been a great conversation. Yeah, it's been a great conversation. Yeah. This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building.

58:51With Tasty Trade, you can trade stocks, options, futures, and crypto all in one platform with low commissions, including zero commissions on stocks and crypto. so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One-Click Trading, and Smart Order Tracking.

59:42And if you're still learning, Tasty Trade offers dozens of free educational courses, plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker-dealer and member of FINLA, NFA, and SIPC.