Imbue CEO Kanjun Qiu on Transforming AI Agents Into Personal Collaborators - Ep. 239

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. One of the big transformations being enabled by AI is the way we create software. From coding co-pilots to in-development systems built to translate plain language requests into fully functional applications, generative AI is fueling a new wave of tools to help us create software faster. Our guest today is Kenjun Q. Kenjun is co-founder and CEO of Imbue, a three and a half year old company somewhere in there there founded in 2021, that is building AI agents that work with us to translate our ideas into code and bring them to life.

0:46There's a lot more to it than that, but why hear it from me when you can hear it directly from Kanjun. Kanjun Q, welcome. Thank you so much for taking the time to join the NVIDIA AI podcast. Thank you, Noah. It's great to be here. So let's talk about software development and AI and Imbue and all of that good stuff. But maybe let's start with agents. Agents are kind of a, I mean, I don't want to use the word hot because I don't want it to sound, you know, fluffy, right? But agents are a thing right now. We've had some guests on recently talking about agents in different contexts. And Imbue's approach to agents is something worth delving into.

1:22So maybe we can start there. What are AI agents? Why do we need them? Why isn't Imbue working on them? Yeah, agents are really popular right now. So Imbue was founded in 2021, early 2021. And at that time, our goal was to figure out how to make and work on general AI agents. At that time, people thought we were totally crazy. Like, what are agents? Everyone's working on AGI. You know, AGI is going to rule everything. But what we were really interested in is how can we have systems that we as people kind of can mold, shape to what we're wanting, as opposed to, oh, this external intelligence that knows all the answers.

1:59And so we started as a research lab at that time because the technology was certainly not good enough to build general agents that could reliably do anything that we wanted them to do. And, you know, in the very beginning, we thought of agents in a similar way to how a lot of people think about agents today as these kind of, you know, think about what an agent is. Often people think about kind of an autonomous personal assistant type thing. You ask it to do something, it does it for you, it comes back to you. Now everyone has their own personal assistant. And actually, a lot of our learning has been that that's a really tricky user experience.

2:35And we experience it ourselves with human agents, which is that often when I delegate something, it comes back. It's not quite what I wanted. And now I have to negotiate how to get what I wanted. I was listening. I told you before we recorded, I was listening to a little bit of your fireside chat with Brian Cotanzaro from GTC this year, which listeners go check that out. It's a great listen. And you were talking about the difficulty, and I can relate to this so much, the difficulty inherent in delegating work to someone else, right? And to your point, thinking of it as humans, you have to break the problem down, you have to sort of figure out, well, how much do I tell them exactly what to do?

3:14Yeah. All that, yeah, yeah. What context does it need ahead of time? What instruction should I give? Delegation is actually a really tricky paradigm because it actually puts all the onus on the person who's delegating to define the problem, define the scope, Of course, the person who's being delegated to the agent might come back with some questions and stuff like that. But it's a very tricky thing to trust. So what we've learned over the years working on general agents is we've actually started to think about agents in a very different way. And this is both from a kind of pragmatic business, like user perspective, and also from a mission perspective.

3:46The way that we think about agents is if you think about what an agent is, what this personal assistant is doing, What it is, is it's kind of this intermediary layer between you and your computer. And you're kind of telling it stuff or interfacing with it in some way, whether it's a UI or a natural language. And it's interfacing with your computer. And the most effective way to interface with your computer is by writing code. That's what your computer is made of. And there are really kind of two types of code. There's like everything is hard-coded, which is the default for software today. Everything is hard-coded.

4:20So now your agent can only do a subset of things. or now with language models, you can generate code. You can write code that does new stuff that's not hard-coded and now you have an agent that's much more general. It's able to do sets of tasks that, you know, I didn't program into it ahead of time, but now it's able to do. And so the way that we think about what agents are is actually as this intermediary layer between your new computer and it's essentially a layer of abstraction on top of programming that allows us as regular people to be able to program our computers without even thinking about what we're doing as programming.

4:53And so we think of our mission as essentially trying to reinvent the personal computer and really deeply empower everyone to be able to create in this computing medium of the future, because this digital medium is becoming more and more important in our lives. And what we want is actually not what I want, at least personally, is actually not a super centralized assistant that everyone, someone else has decided is able to do this, integrate with that. What I actually want is something that I can make and own that is mine and that does what I wanted to do, kind of serves me. And today we're kind of in a world where like all of our software is rented, it serves other people.

5:32So that's kind of what we think of what agents are is this layer of abstraction on top of programming makes it so that it's very intuitive for everyone to program. And that actually requires quite a bit of invention. So can get into that and historical point out. Yeah. Well, the way you were describing it, I know you weren't describing the sort of AI, the assistant, the agent layer, all the A words, AI, assistant, agent. You weren't describing the agent layer as replacement for a user interface. So you mentioned, you know, UI, but that's kind of what I was thinking of, right? That abstractive layer that's kind of in between.

6:02So how is Imbue approaching it sort of on the ground and working with, you work primarily with enterprise clients? No, we're primarily with, I would say it's prosumer. So people who are, so the way to think about what we're doing is instead of thinking of agents as automation systems, right now we're in kind of the agent automation agent paradigm. We think of agents as collaborative systems. So how do we enable a system that empowers me and helps me do more of what I want to do and work with, I can work with it. And so in the beginning, we're actually, you know, enabling you to write code. Right now, these models are not that good at writing code.

6:38As they write code, you actually have to go check it. So you have to understand how to write code to go see, oh, did the agent do a good job? Right. But over time, as these models get better at writing code, now you don't have to check it anymore. And so in the beginning, we start with software engineers as a prime, or we call them software builders. So you don't have to be an engineer. I'm a software builder. I'm not a software engineer. We start with software builders who can. I'm a prototype builder then. I wouldn't go as far as software. Okay. So you could probably be a soon-to-be user. once we get to a place where you don't have to read and write, touch the code so much.

7:10Right now, we're targeting software builders who can read and touch the code and be like, okay, well, it's not quite right. We'll adjust it in this way. And over time, as the models get better and you don't have to be so low level in the code, now more and more types of creatives, types of builders can use these systems to really mold and shape their computer to what they want it to be. And what level of complexity, what level of depth are, you know, is the actual software that users are building within Vue. You know, there's the issue of accuracy, obviously, as you were saying, they're not, you know, none of the models are creating 100 % perfect code yet.

7:45But I also wonder how complex can these things get? And that kind of brings us to talking about reasoning. We don't have to get there yet, but kind of edging towards that conversation. Yeah, I think one of actually our biggest learnings has been that if as a user, I have a code base that is fairly modular and I've kind of broken things down, then the model is actually pretty good at dealing with a very complex system because it doesn't have to load that much stuff into its head and it doesn't have to like cross check all of these dependencies. So just like with humans building software where you don't want a ton of dependencies, also, you know, if you have a slightly more isolated system, it'll do a better job.

8:20Similarly, there's a lot of kind of user expertise in using this kind of product. So our product, it feels very collaborative. It's almost like a document editor and kind of like interfacing and interacting with an agent in that way. And so as a user, you can basically, we learn to give it tasks that's more likely to succeed out. And we learn to structure our work so that we can delegate it to agents. And we've seen this with other AI tools as well, like Copilot. Our team definitely writes code in a slightly different way so that Copilot can work well for them. Right, right. To your question of complexity, it like really depends on the user.

8:54Like some, you know, some of us can make it work with really complex things. Yeah, yeah. Where are you seeing agents being used the most? Or perhaps, you know, it's more that they're having a dramatic impact where they are being used. And how does that translate into businesses, you know, remaining competitive, having a competitive edge? Folks that I've talked to and I keep talking about, you know, 2022, last year, we're kind of the year of the models coming out and mainstream taking notice of Gen AI. And perhaps this year has been the year where people are trying to figure out what apps do I build to leverage these things, you know, as an app, as a software building business or as a business that does other things that wants to leverage this.

9:34So where is in view seeing impact being made or even, you know, looking to areas in the near future? You know, the interesting thing about agents is it's such an ill-defined term right now. And people are calling all sorts of very trivial things agents, and that's fine. But I think there's a spectrum of kind of agent usefulness, effectiveness. There's like a system that scrapes the web and then aggregates the data and pulls the data out in some way. This is kind of like basically just a Gen AI model. Like, you know, it's kind of very similar to ChatGPT, but you like put something on top of it to like dump the output into a different system.

10:08You could call that an agent. So some people call that an agent. We see that kind of thing being implemented in all sorts of places. But I think the much more exciting thing when it comes to agents is these more general agents that enable people to kind of start doing things that they previously didn't even imagine that they could do. You know, I think like some of the really simple examples right now are for us, like some researcher or scientist, a biologist has a ton of data that they need to process and they're not a software engineer, but they're technical enough that they can kind of like pull in the data and then get something out of it, get some analysis out of it.

10:43If they're able to use something like this, that lets them work at this slightly higher level. Or, you know, kind of over time, a very exciting thing is as we start to build the tools that we need, like an example is my grandmother gets a bunch of scam calls in Chinese. But all of her calls are in Chinese. And if I want to build a piece of software that filters out her scam calls from her other calls, like this is very hard right now, even for me as someone who knows how to build software. And it's such a niche market. Like no one else is going to build that software for her. We've tried to find software like that in the U.S.

11:18doesn't really exist. And so, exactly. So right now we're in this world where software is built by... Not to interrupt you, if it exists in the U.S. for English language spam, it doesn't work that well either for my... Exactly. Exactly. So, you know, right now we're in this world where other people build software for us. We have to rely on other people to build software for us. And it's actually really strange. Like we don't really own our digital environments in that way. Like everything is kind of built by someone else because it's too hard for us to like build our own things. And I think there is a future in which like I could actually pretty easily build something for my grandmother or for my community or for my group of friends or for my church to manage registrations or whatever it is.

12:00Right. And that can be really tailored to my particular use case and to me, my community, my friends. And so I think the really exciting thing about this future is like all of this like bespoke software as opposed to today where we have this kind of centralized software. It's almost like people don't often think of their digital environment in this way, but the digital environment is like the physical environment. And today it's as if we all live in corporate housing. I used to be so excited that I could listen to any music I wanted for, you know, 10 bucks a month. And now I'm thinking like, but I don't own any of it.

12:34They could take it away from me in a second. Yeah. And honestly, I think a lot of the kind of frustration people have about big tech, about technology is that we don't feel like, and I don't feel like I have control over these things that are a huge part of my life. And so that's what at Imbue, what we want to do is give that control and power back to the people. And we do that by creating these tools and systems that collaborate with you to help you be able to create stuff for yourself. So how hard is it to build these things for folks to use? What, you know, as you mentioned, there are different, many different, you know, voices, individuals, companies talking about agents, agentic AI, and a lot of them are defining it, talking about it at least slightly different.

13:18I'm sure taking, you know, different approaches kind of under the hood. What are the challenges? What are the things that, you know, we can get a little bit technical here as you like, some of the things, some of the problems that you and your teams are solving to make it easier for the rest of us to translate our ideas into software? Yeah, so some problems I would say are kind of universal across all of these different types of people who are building agents. And some problems are unique to us and what we're trying to do. So I would say, most people are building agents in this kind of like workflow automation paradigm I mentioned earlier.

13:50And so for that paradigm, robustness, reliability is really important. Like, okay, I built a thing that responds to customer service tickets. But if 3 % of the time it says something really terrible to the user, like this is not a usable agent. For us, reliability and robustness is important, but it's actually a little bit less important. As it gets better, the user experience just gets better. As a user, I don't have to check stuff as much. But even if it's not the best, it's still okay. Like I can still use it as the user and I'll like fix the bug that, you know, the model produced and that's okay.

14:22So a lot of what we think of is kind of like, how do we get agents to meet both the model capabilities and also users where they are today? So that expectation is built in that we're not at the stage yet where it's error-free. And as a user, you need to know that. And it's not just like, okay, you have to accept that, but it's actually like your experience is going to wind up being better, right? Because you know you're part in it. And it's, again, as you said, it's not sending it off to do something and, you know, giving us the final result we had no part in. Yeah, exactly. I think, you know, people often think of agents as a research problem, but we think of it as both a research problem and a user experience problem.

14:58And this user experience part is really about setting the right expectations so that with the experience, so that it's like, I'm not expecting it to go off on its own for 10 minutes or 10 hours or 10 days and come back with something magically correct. Instead, I'm kind of iteratively working with it and seeing, oh, it's kind of like clay. You know, I'm molding it, shaping it, shaping the output. But I think the workflow automation agents, some of these agents kind of, they're a little bit more, the bar is higher for how accurate they have to be. Because what we have found is that as a user, the longer it takes to come back with an answer, the more I expect the answer to be correct.

15:32And I'll get frustrated if it takes a really long time. So we're very much on the like highly interactive, don't take a super long time to come back with something. Be an agent that really works with the user side. Right. Thinking about the idea of moving from, I'm expecting from a computer, right? The utmost inaccuracy, my calculator always says two plus two is four. Kind of moving from that to just a different frame of mind as an end user saying, okay, we're prioritizing, you know, I don't want to put words in your mouth, but kind of the speed and experience and you know going in, it's not going to get them all right.

16:04Is that something that, you know, because this is the nature of AI and Gen AI and people are kind of used to that by now, people are accepting of? Or is there still kind of a, I don't know, maybe I'm just old, is there still kind of a mental hurdle to getting past that expectation? Yeah. So one of our core philosophies is that we need to meet people where they are in terms of what mental models we have in our heads as people. And so actually a good historical analogy is back before the personal computer, people were really excited about the supercomputer. And the first personal computer first came out, everyone made fun of it.

16:36They They were like, this is a hobbyist toy. And the supercomputer, you know, you accessed it with a terminal. You were time sharing on these supercomputers. It was not especially usable. And so a very small set of people were able to use it. But as time went on, a small group of people at Xerox PARC actually invented a lot of these primitives that led the personal computer to be usable by people. They invented the desktop files, folders. These are like concepts that we understood as humans at that time. And so for us, you know, part of actually building a good user experience around agents requires invention.

17:08It requires inventing concepts that match kind of like are able to map this technology to what we currently understand as humans. So earlier I was saying it's kind of like a document editor right now, you know, our current product experience. And it may not ultimately be that way. But a document is something that I as a person today understand how to edit and work with. And it's almost like an interactive editor that helps me accomplish my tasks. So to your question of how users receive it, one thing that's really interesting that we observe is software builders, the feedback so far has been, wow, like this is really interesting.

17:42It lets me work at this higher level. I don't have to dig so far into the code all the time. I can kind of stay and thinking at this higher level. And it's actually able to go down into the code and surface to me, like do the task, and then I check it. And that's pretty cool. It like lets me move a lot faster. And that's really interesting. You know, that for us, that's kind of the primary thing. Like I want people to be able to work at our human problem level of abstraction with software instead of having to get so deep into the weeds. Yeah, yeah. No, I can relate from the standpoint of learning how to use these tools when I'm writing, when I'm not.

18:24Yeah. On the podcast, asking people like you questions. You know, a lot of my work is writing. And if I'm working from a document, a transcript, source materials, it's that same thing when I can use the tool to just kind of surface to me, you know, did anywhere in the document or in the transcript, did Kanjun talk about pineapple on pizza? You know, and just it being able to service that back, right? It saves all the time of going through the document. I don't need the exact words. I don't need it 100 % then. We'll get to that later. But I can just kind of go back and check like, oh, right. She said she is, isn't a fan of Pep Run.

19:00You know, yeah. And it's incredibly helpful. You know, it's not just a saving time, but I think you said it really well. It allows you to stay at that level of thinking. Exactly. Yeah, I think our core, the thing I really care about is helping people be able to be creative in this way. And often there's such a barrier between our ideas and the execution of those ideas that we never get to do a lot of things that we really want to do with our lives. And so I think the true power of computing as a medium hasn't really been unleashed yet. And we want to unleash it. And what that looks like is that people are able to kind of take their ideas.

19:31You can take your ideas about writing this piece or aggregating all of the pieces you've written and being able to draw insights out of them in order to create your book. like take these ideas and actually be able to work with them at this higher level so that you're not always bogged down in the weeds. And I think the true power of AI, the true power of computers, like that's what it enables. And we're not there yet, but we can get there. And it's not just about automation or business automation, business workflow automation. Right, right. Now, what was the, in your conversation with Brian from GTC, what was it that he said, I've got all these ideas and I sit down to code and I'm like, import?

20:03What do I want to import? Right. And I think that was great because you're derailed immediately and I can relate in the work that I do that, you know, yeah, yeah. Yeah, 100%. Yeah. One of our users said, wow, I never realized how much context switching I do when I'm writing code from high to low level. Same with when you're writing, writing normal stuff. I'm speaking with Kanjun Q. Kanjun is co-founder and CEO of Imbue. And we have been talking about Imbue's work on AI agents that help people code and really fascinating approach that, you know, as we've been talking about, I think goes beyond just expressing in code, but code being the way that we interface with computers and get them to do the things we want them to do.

20:44Well, I want to ask you about AI models and reasoning. And then I also want to ask you sort of about scale and what goes into, you know, building an agent for yourself. And then what goes into building agents and multiple agents and agents that collaborate for users at scale? Is there an order we should go in? Should we talk about reasoning first? Is there a relation? That's interesting. Let's talk about scale first. Okay, cool. Yeah. So one way that people think about scale with agents is a lot of agents interacting with each other and kind of what that looks like. And some people do that by giving different agents different prompts so they have different personalities and things like that.

21:24And honestly, I think that's a little bit, it's interesting. It's a little bit limited because we already have agents today. All software is agentic. The whole point of what an agent is, is it's something that takes action. And it uses your computer to kind of like execute something. And so almost all software is executing something. It's kind of like changing the state of your computer, website, data, etc. Now, the difference between most software and like AI agents, what we call AI agents today, is that AI agents can process stuff in a way that's like not fully deterministic. But even so, we still had AI agents in Facebook newsfeed, for example.

22:04Recommendation Engine is an agent that non-deterministically decides what to show you. So we've had agents since forever, since we had software. And so, you know, kind of the way I think about scale and agents is actually about, is the same as the way I think about scale of software. So in the future, next 10 years, I think there's going to be this explosion of software and the software is going to be slightly less hard-coded than before. It's going to be able to work with more ambiguous input. It might be more interactive. Hopefully, a lot of people are going to be able to create it if we succeed.

22:33And so now we end up with this giant ecosystem world of software that's far beyond what we have today. And in that world, what happens is now you have a lot of different automated systems interacting with each other. And this actually could be super exciting. Every person could have their own automated systems that do things for themselves, for their lives. They have, you know, I'm supported by the software that surrounds me, as opposed to today, maybe being bothered by it. That's funny. I was listening to you and I was thinking about how to phrase, how to try to phrase the question, getting back to your point about, you know, sort of, I was thinking of it as sort of one size fits all software, you know, that's deterministic It does what it does versus, you know, me being able to create and reshape things as we go.

23:20And you answered it for me. So that's great. I love this. I love this one size fits all software today versus future bespoke software. That's a great term. Are you worried at all about, I think, I don't know if the term AI slop applies to code, but this idea of, you know, AI models creating text that's kind of meaningless or valueless, but it's being automatically put out onto the web and stuff. From a very sort of, you know, relatively ignorant point of view, the notion of model-generated software that can run itself being deployed out on the public web, you know, is a little scary to me. But also, I'm sure there's stuff out there.