How Anyone Can Build Meaningful AI Without Code - Ep. 283

0:10Hello, welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Before we begin, a quick reminder, if you're enjoying the podcast, take a second to follow us wherever you get your podcasts. It helps us out, and it helps you out by making sure you never miss an episode that just show up in your feed. My guest today is Shanae Levin. Shanae is the co-founder and CEO of Impromptu AI. They're focused on helping non-technical folks create AI products, which is, as Shanae and I was just saying before we hit record, it's where a lot of focus is these days. So this is going to be a great conversation.

0:45And let's get into it. Shania, thank you for joining the podcast. So happy to have you. Thank you so much for having me. It's been such an honor. So let's start with a little bit about your background, if you don't mind, and then getting into what inspired you to co-found and build Impromptu and tell us what it's all about. Absolutely. So background is I studied business and computer science. And then I spent some time at Google working on developer tools for Google Home and Android, helping build Android applications for millions of developers all over the world. And then I spent some time at eBay working on both ads and more traditional machine learning, which was super fun.

1:26And then I got tired of just working at big companies. So I was like, I'm going to try this startup thing. And I then went on to Cloudflare, and then I was a senior director of product at Docker. And then I was head of product at a Series C company, startup company. And then I decided to go out on my own. I built a company called CodeC, which was helping developers to master the understanding of their code bases, which back in 2019 was revolutionary. it's like the uh the bt before transformer era exactly i've been doing this a really long time helping people to understand their code and uh we built something called code cai which basically allowed you to chat with your code base so i was in the you know gpt2 beta like way early in the process and so we were building um this generative system back then and then codec got acquired and And I decided to take some time off and travel around the world.

2:28Fantastic. I ended up really starting off building just a very simple app with Lovable. Tried to do that. And of course, being the technical person that I am, I really broke it. I just full on broke it. And then I put a pause on that. I ended up taking a role for a short amount of time at a subsidiary of Fox Sports to help them with their AI. And right around the same time, I ended up meeting Sean, Dr. Sean Robinson, who was my co-founder. And he ended up saying, I invented this thing to get up to 98 % accurate outputs out of AI. And I looked at him and I said, what? Wait, what? Like, you can get up to 98 % accurate outputs out of AI.

3:22Like, right, it's really hard to get AI outputs to be accurate. And that is the next frontier. And so we decided to really team up and build and prompt you together. He is a computational physicist and a researcher. And we're always inventing new things. And it's been really one of the highlights of my entire life to be able to build this company for people. That's incredible. I have 9 million questions about your background that we don't have time for right now. So I'm going to narrow it down to just one. when you took time off and traveled? Uh-huh. Favorite spot? Oh, my God. You don't have to pick one.

3:55Okay, okay. I'm going to pick two. Okay. So it's really hard, but I'm going to say I loved Japan. Yeah. Japan is amazing. And my husband convinced us to spend three nights in the Sahara Desert or like traveling from Fez to Morocco. And then we spent a night in the Sahara. And I'm definitely old enough to remember, you know, Windows 95 screensavers. It looks exactly like that. Like in real life, like a screensaver comes to life. So when you're done founding and selling successful startups, you can write some travel memoirs and link them back to the tech experience. It is great. Exactly. So with all of your technical background, what made you, I mean, I don't know what made you, it makes sense to me why you wanted to found something for less technical people to be able to create products, kind of get a handle on things.

4:51But sort of, was that, I don't know, when you met Sean and the two of you decided to do this, was that the initial idea or did you kind of iterate on some stuff before landing on Impromptu? Yeah, no, actually that was not the original idea. We had no intentions on starting a huge company. We actually were started off by using the tech that he invented, the art, which we call now our AI core, which is our optimization tech. And we started just building AI apps for people. Like people needed our help to build these AI applications. And we were like, hey, how do we help them do it faster and more accurately?

5:30And that was always the goal. Like how do we make sure that we get good outputs. I mean, because back then, like, you would get, you know, it would hallucinate like crazy. I mean, it kind of still does, but like, it was wild, the responses. And so one of the things that we need to do is really build trust with AI. And so no, we were just manually going and helping people in our network build AI applications. And then after, you know, over time, we ended up building a bunch of these. And we kind of took a step back. I I actually literally have this recorded the moment that this happened. We're like, oh, the infrastructure to build AI applications across all of our customers is exactly the same.

6:13I wonder if we can agentically write this AI application. And Sean looks at me and goes, wait, you want an AI that builds AI? And I was like, well, is that possible? And he's like, no, it's not possible. And then he does the thing that he always does where he says no. And then, you know, 20 minutes later, he goes, well, maybe. Right, right. Maybe if we did this and this and this. And fast forward till today, and here we are. We have an AI that is very specialized, specialized builder to build actual AI applications. And so you're working primarily, your customers are primarily or entirely enterprise-sized clients?

6:55Yeah, so they're all businesses. businesses of different sizes, right? So they're mostly large enterprises and mid-sized enterprises. Typically, if you have a SaaS application already, right, you know, you have a platform, you have a service that you're doing with a website, or you have, you know, a running business, how do I transform that into an AI native company, right? And that's really where the gap is, right? Like, you don't, you need to start from this legacy code base. I was going to say that's the trillion dollar question, except, I don't know, I keep thinking about the 98 % accuracy and that might be the two trillion dollar question.

7:34Well, here's the thing. Both of those have to work together. Yeah, exactly. You can't have, you know, it doesn't matter if you can get 98 % accuracy if you have to throw out your entire code base, right? And it doesn't matter if you, you know, have to, you know, you're starting from a greenfield project and then your output kind of sucks, right? So you need a lot of these pieces to work together. And we've invented five new pieces of tech from adaptive context engines to infinite memory to being able to our optimization core to be able to ingest a code base and say, hey, I'm going to import this whole code base from GitHub and now add AI to it.

8:14That wasn't possible before we invented the things that we did. Right. To kind of focus on the enterprise scale for a minute here, when you're working with a client of that size and complexity and teams of data scientists and so many stakeholders, and you start working with somebody who's less technical, but the idea is to help them build something kind of in that context, how do you approach it? And I don't know if the best way is to kind of ask you questions piece by piece or if you want to walk through kind of an example project. But when I'm messing around with AI and having it build something for my own purposes, right?

8:51I mean, obviously, the threshold for success is much lower. It doesn't matter if it's not perfect just for me. But also the process, you know, I can just kind of put my head down and mess around so I come up with. I'm at work. I'm in the team. It's a very different situation, right? So what is that like for Impromptu and the folks you're working with? Well, it's multifold, right? Because the mission of our company is to make AI accurate and accessible, right? So how do we make sure to get this technology into the hands of people who, it's not that, you know, they haven't coded before, because I talked to 20-year developers, like people have been developing for 20 years, and this is a new technology that we're all learning at the same time, right?

9:34Right. And so all of that comes with education, all of that. It's like if I said, hey, you're going to go walk into Sephora and now you need to go and find sunscreen amongst a thousand products. How do you do that? You know what I mean? So like it's just. I do. I tried to get something from Sephora once for somebody else. It's hard. You're lost. There's a lot going on. Exactly. It's the same thing. Right. So it's not a less technical person per se, right? But it's someone who's less versed in what's happening in AI, which is a race to keep up with. Exactly. And so we definitely think about like, hey, everybody is learning generative for the first time.

10:17What are the pieces to educate anybody at any technical level? And so we do that not only with our UI, where the UI of the builder actually knows what all seven pieces are to build, the full seven pieces to make an AI application, but it asks you back and forth questions to make sure that you have supplied it with the right information. And it goes and has a full conversation with you. And we also have a co-build model, right? So we as experts will also alongside of our AI tech also build in collaboration with the organization and educate their team, work alongside their teams so that, you know, it doesn't feel like we're just going to kind of leave you in lurch.

11:02Right. Right. And so because enterprises, we need like everything is changing so fast. Right. You need to have someone who's keeping up with this while you're running your business and running your team and running, you know, all of the things internally. Someone needs to keep up with this stuff. And that's basically us. Yeah. What is that? You know, we I use the word transformation all the time. You know, you just mentioned it and it's accurate, but what does it mean in this context of, you know, transforming an industry, transforming a business and the work that you're doing sort of, you know, on the ground level, right?

11:38I mean, you've got that high perspective from your background and everything, but you're right there doing it. So, you know, folks like me are looking at kind of broader trends and thinking like, wow, you know, the ability to, AI has really changed and it was really changing the way search works, right? Yeah. It's just a giant example. Right. Right, kind of a big glossy thing to say. But what are you seeing? Are there certain trends that are emerging, certain types of problems that people are having more success or less success with? Yeah, so the larger trends as we're kind of working on these things together is a huge thing in physical AI, right?

12:18Like how do you move things in a physical space? We work with a ton of folks in that capacity. We also are really thinking about context, right? And memory. And how, as you think about, like, if code is going to be this, you know, arbitrary commodity, what then becomes important? And then that's the data, right? How do you then tackle this next frontier to create an end-to-end solution? So now on, you know, last week, we announced our custom data models in straight into our builder. So now you have not only you can build an application, but it's built right off of the custom data and it's trained in the exact way that you would want it to be trained.

13:00And so you get really highly accurate responses. Right. And then that becomes really, you take it one level up. So you start thinking about multi-tenancy. You start thinking about specialized models. You start thinking about AI doing decision-making for you. And that's really where you get into this thing called provable AI. Like what's actually happening? What decisions is it making? How do I control? And then thinking about the governance plane. Well, if I have 10 applications across my organization, how do I make sure that it follows these rules? And we're tackling all of those things in one fell swoop to be able to make sure that if you are going to go down this journey, that it is not only accessible to anyone on your team and it scales appropriately, but it also feels like you have full control over it.

13:47Right. How big is the gap for a first-time generative AI builder, to put it that way, from something that I might build and say like, oh, this seems to work in my use case, going from that to something that's production-ready? How big is that gap and how do you help people bridge it? Yeah. So great question. We actually think about it slightly differently where I can get you a chat bot in probably 10 minutes. It has all the production ready pieces that you need. Right. But what you actually are now, what we're thinking about is enterprise scale. How do you do that with terabytes of data? How do you do that in real time across millions of users?

14:32Right. How do you do that, you know, where you have full governance and control? Right. So like today in our self-serve platform, if you're on like a big enough plan, I can get you very fast, very accurate system in probably 10 minutes. But it's the bridging the gap as you grow and have multiple interconnected AI workflows and, you know, being able to make sure that they're all accurate and all running to the best of their ability. Right. You used the word accessibility a minute ago and, you know, talking about AI being accessible to first-timers, less technical people. What does that mean, you know, kind of day-to-day in your work?

15:14And maybe, like, are there ways that people use their kind of misconceptions people have about that term? Yeah. So I think AI accessibility at this stage is allowing anyone with an idea, not just to use AI to build old technologies, right? It's not just using generative to make HTML, CSS, JavaScript. Like all of those are great, right? We absolutely need those. But how do we use AI to actually build AI, to put generative, thinking, living, breathing applications into people's hands? And so it's not only new for developers, but it's new for the business owners. It's new when you're thinking about companies being disrupted.

16:00It's new for the venture community. It's new for everyone. And so how do we democratize and actually have this new technology to be disseminated to anyone who wants it? Because now, you know, we have customers who, you know, are doing these amazing things and now can enable these even more amazing things for their customers. So, you know, we have a mom and daughter team who is, you know, building financial literacy, or we have a CPG brand who takes ocean plastic and turns that into activewear, right? Like, so not just the super technical large enterprise companies, but like every across the spectrum.

16:46The example of the mom and daughter team, like, I can wrap my head around that easily because I've, you know, typed in, like, help me code a dashboard for podcast analytics, right? Yeah. And I've gone on that journey. How does AI help? And, you know, impromptu, your experience, how did it help the process of recycling ocean waste to create activeware? So really thinking about how do you bring all of these systems together, right? How do you make recommendations to the founders and their teams internally to make sure that if you're spinning up an event to collect the bottles, how do I spin up a new app, a new city across the country to do the exact same thing?

17:33How do I operationalize this process? Tell me how to do it, AI. But in order to do that, that's very custom data. right that you're not just going to get from a general model right you need their system and their models um because that's not a very typical use case and so those are the sorts of things that now you enable i use ai every single day uh in my in my system to build this company um it's unheard of going from five years ago building a company then till now like my day-to-day is completely different. Yeah. Well, it's funny, as you were saying about when you and Sean, I want to call him Dr.

18:15Sean. You should call him Dr. Sean. When you and Dr. Sean, when you and your co-founder got together, you know, in the question of, wait, you want me to build AI that builds AI? And like you said, fast forward, and now, you know, I mean, I heard Jensen say it, but lots of people are saying it, that AI is going to mostly generate tokens to be used by agents, other AI systems, right? The majority of the tokens as we go forward are not going to be creating output that humans see, but that another AI is going to act with. So it's amazing. I'm sorry. Yeah, no, it's truly amazing. We're working with some customers and some people's dreams who would have never had the ability to do this.

19:04But then also you're thinking about like, Like, oh, actually, I don't, like, eventually, we're not going to need a full human into this at all, right? And so we can actually AI generate human in the loop systems, but we can also just generate a full end-to-end API that, you know, goes out from the front end, does a whole system, does an AI generate and puts it back as an API back into the customer system. And, like, no one ever knows that we're doing anything, right? Because it's fully containerized and all of those things. So we're already seeing that day to day. My guest is Shania Levin. Shania is CEO and co-founder of Impromptu with an E, E-M-P-R-O-M-P-T-U dot AI.

19:49I want to get that out there if you're listening now and want to look it up later. And we're talking about Impromptu's work helping people of all technical abilities, but certainly people who are less used to building with generative AI, build products at enterprise scale and production ready and the whole thing. So I want to ask you kind of a more technical question here. Impromptu uses NVIDIA CUDA? Yeah. Can you talk about how the CUDA libraries for embedding and classification in particular improve your performance and make AI efficient, all that good stuff, but how it actually translates to the work that you're doing and then in particular the work that your customers are doing in creating with these tools?

20:33Yeah, so we consider an impromptu to be a mixed code builder instead of no code or pro code. It's kind of, we call it a mixed code. Under the hood, though, you know, parts of it are essentially, you know, a big embedding and classification machine, right? So every time, you know, we map a new feature or iterator design or, you know, you search your custom data model, you know, we're turning that into vectors and making bigger decisions on top of that. Now, we have our own kind of custom models and our own secret toss alongside of that. But we use NVIDIA's code libraries so that all of the heavy math natively runs on GPUs instead of slowly on CPUs.

21:14And that really gives us two big wins, right? First, it's performance. So like all of the everyday creators in our UI get that instant feedback. They can request a feature or upload or tweak a prompt. And the AI just kind of responds. And so they can iterate much faster, ship much more useful products. And then the second thing is efficiency, right? So CUDA lets us serve a ton of those workloads in a relatively small GPU footprint, whether that's in our cloud or our customer's VPC. Right. And how does CUDA help? I mean, you talked about this a little bit to put a point on it. How does CUDA help kind of bridge the gap?

21:55And I'm wondering, from your perspective, how big or small that gap really is these days between kind of the cutting edge AI research that's being done and then the tools that, you know, your customers are actually able to use to build things. Yeah, I think at this point, you know, the gap is mostly filled. If I do say so myself. You know, you don't, right, today, you don't need, you know, a big machine learning team to build something sophisticated. you know, CUDA kind of helps us with that hard math. And so our platform turns all of that into, you know, this fast, affordable, you know, just works experience for, you know, non-technical and technical teams.

22:37Because like technical versus non-technical, those now are, they go together now, right? It's generative or not generative, right? And so as we're thinking about that, Like all of those things I've now been able to do it in a, you know, a self-serve if you really want to do that. Or if you want the pros to actually build it for you, you can just throw that out, right? We can actually just use our own systems to build it on your behalf and do the integration for you. But all of the things under the hood are basically all like taken care of. All right. So this is the point where I come back to this 98 % thing that, you know, I was like, I don't want to ask up front.

23:18because probably it's secret sauce and we're not going to get too into it. But can you get into any of the details of how it works? Yeah. So it does work a little bit differently than most other systems. And what we found is that a lot of people talk about benchmarks, model benchmarks, right? Sure. Model benchmarks are great. Everybody has a model benchmark. Excellent. But how do you translate that model benchmark to the everyday enterprise who's like, I have a goal. I have things to do. I have stuff to do. I have stuff to do, right? And so our accuracy is actually redefining that success, which is task success.

23:56And because we work with so many use cases and so many customers and enterprises, we allow the user to define what success means to them, what that task is, what success looks like. and we will use our full optimization engine that optimizes the entire system, including the model, the data, the prompts, the evals, all of the under secret sauce under the hood to optimize in real time towards that goal that the user sets. And we measure it directly in our dashboard. So you'll see a big number of how we improve that accuracy towards the goal that the user sets. The other thing that might not be so obvious is we have two modes.

24:38One is for, it's called manual optimization, which are all for all of those technical, deep technical developers who want to tinker and turn knobs just like myself, right? I want to do that. But you can actually see the output towards that accuracy and it will actually give you suggestions. You can click the button however many times you want to click it and improve it however many times you want. After about 30 runs though, we can turn on something called automatic optimization. So if you're less technical and you're like, I don't know what the heck that is, that sounds smart. It is. But if you don't want to tinker with it yourself, we'll just do the automatic optimization for you up to, you know, 98 % of that task accuracy.

25:23And we're always improving it. It's always, you know, we're always trying to get to, you know, three nines, which I definitely presume that we will one day. And so that, you know, that speaks to, and you brought this up before, and obviously it's a big, a big thing right now in the industry and the marketplace is building trust. Definitely. Are your, you know, your customers, like, are they looking at, like, does that dashboard, that figure of optimization, does that do a lot to improve the trust? Is it really just seeing the results and being able to say, like, yes, it helped me get this project, you know, off the ground and running and it's working well?

26:01How do you build trust right now when there's kind of this tension between the race to build out all the infrastructure and the data centers and everybody wants chips and can't get them, and then everything that we've been talking about? Yeah. So we subscribe to something called provable AI. And we think that that is the absolute way to build trust, right? And make it so that it is accessible, right? If you can't see it, feel it, touch it in a theoretical sense, because it's all bites and bits, it's all ones and zeros, right? you know until we get to physical like you mentioned until right exactly so uh thinking about that you know seeing okay what decisions is the ai making how do i roll back right how do i you know if i have infinite memory you know how like what you know is involved in this decision right like if i'm doing a custom data model you know how what data is kind of going in being able to showcase, like here are all the runs that we took, right?

27:10Showing those accuracy numbers in the dashboard, right? We always try to showcase, you know, provable, hey, this is the number, this is the suggestion, this is the, you know, the set that you need to be able to do. Now, can you disconnect those things for privacy concerns? Absolutely. But proving to the user that we're doing what we say we're doing is of the utmost importance. Yeah, yeah. To switch gears a little bit, as a woman in tech, right, which like everything else, particularly everything else in tech, has changed a lot very quickly in certain ways. And not to speak for you, but probably not so much in other ways.

27:55But then, you know, working now with a lot of people who are, you know, the whole idea of making AI accessible, right? How does your experience as you, as a woman in tech, you know, co-founding and running this company now and working with folks, how does that inform kind of how you see, you know, barriers to entry? Whether it's, oh, I'm new to generative and I want to start working with generative. Or, you know, oh, I've never done something at production scale. Or, oh, like, I'm a woman and I've experienced what I've experienced. And so I'm interested in this stuff, but I don't know. Yeah, it definitely really, you know, hits my heart.

28:39I can tell you a very quick story. Please. When I, I studied, obviously studied business computer science, but I had a small web development company that I started when I was 19. And then my first big job, like big, big job after I was an analyst was working at Google. And I was like, what? And so I had two choices, you know, go to search, Google search at the time, and then go to this thing called developer tools. And at the time, I was like, I just seen the internship, which I know kind of, but it felt like this really overwhelming technical, like, I'm going to be working with some of the biggest, smartest developers that, you know, ever.