#189 Rahul Sonwalkar: The Future of Programming, AI-Powered Code Generation with Julius AI

0:00We would love to support anything that's possible with code, and I think that's the future. You know, the future is there will be 7 billion programmers and they will all program with English, whether it be scraping, whether it be making software, whether it be making apps, websites or doing data science and statistics. All of us will be programmers because we'll be programming with English. The reason Julius exists and the reason Julius even has a chance of success and the reason thousands of people use Julius every day is because we're obsessed about two things. One is building an AI-native product, and the second thing is focusing on accuracy.

0:34Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I speak with Rahul Solmacher, the founder and CEO of Julius AI. Julius is an AI-powered data scientist that helps users analyze data sets, create visualizations, and gain insights using natural language. Rahul shares his journey from learning to code to building Julius, which has already amassed over a half million users. We discuss the challenges of AI code generation, how Julius compares to other AI coding tools, and the future of programming with AI assistants. I hope you find the conversation as amazing as I did. Hi, I wanted to jump in and give a shout out to our sponsor this week, TrialKey.ai, the market leader, AI-driven clinical trial design optimization and predictor that improves trial design and execution.

1:34I recently had TrialKey on the podcast. I encourage anybody who's interested in clinical trials to go back and listen. With over 90 % accuracy based on precision recall, TrialKey.ai predicts the success of clinical trials more precisely than any other tool on the market. TrialKey.ai contains the world's first clinical trial design simulator, which allows you to manipulate a trial's design to find the best combination for most probable success. TrialKey.ai pulls data from over 350 ,000 clinical trials, inclusive of medical devices, alternative therapies, and non-randomized drug control trials. Pivotal clinical trials often exceed$48 million in cost, yet only 10 % of pivotal trials succeed.

2:32TrialKey aims to address this issue by helping companies improve trial design with AI. A 2011 study found that 60 % of trial protocols needed amendments, with a third preventable through better design planning. On average, completed protocols undergo 2.3 amendments, each causing significant delays and costs around half a million dollars each. AI provides clinical researchers and pharmaceutical companies with in-depth variable analysis and a clinical trial simulator that improves trial design, execution, and drug mechanism of action. Leverage AI-powered clinical trials to accelerate medical breakthroughs at trialkey.ai.

3:20Maximize clinical trial success while minimizing costs. Book a demo at trialkey.ai or sign up for free. Transform your clinical trial outcomes and shape the future of healthcare. Again, schedule a demo at TrialKey, that's T-R-I-A-L-K-E-Y-A-I, or sign up for free. Elevate your trial success and drive innovation in healthcare. Okay, so I usually start by having you introduce yourself, give your educational background, how you got into Julius, how you started Julius. I'm also curious where you got the name Julius. But yeah, so why don't you start by introducing yourself? Totally. So I'm Rahul. I'm the founder and CEO of Julius AI.

4:18Julius is an AI data scientist that helps you analyze data sets, create data visualizations, and get insights from your data with just simple English. We launched Julius about eight months ago, and since then have crossed around half a million users. Over a million and a half data visualizations created. And, you know, as of this morning, users are using Julius, where Julius is writing a million lines of code and executing that code every 48 hours. My story is I moved to the U.S. as a teenager. to study computer science. And I was really excited about the potential of technology, of startups, and how you can impact and change people's lives in really meaningful ways.

5:11So I moved to the US, I moved to Texas, and I went to college at UT Dallas, where I got a scholarship to study computer science. And while I was at the university, I got into the scene of going to hackathons and building projects on the weekends. And through that scene, I was able to meet people who were building things that were technically really hard, but they were primarily building them just because they were really cool. At my first hackathon, I met this guy, and he was trying to write a script to automate the dinosaur game. When you lose your internet on Chrome, there's this dinosaur that shows up and you can play that game.

5:59And he was trying to write a script to just automate that. And to me, that was, first of all, really cool, but also completely pointless. And through that, I got into hacking, writing software, making apps just for the sake of building something really cool. And eventually, I met people who referred me to startups. in Silicon Valley. And I ended up working at companies like Uber, where I did some machine learning on the pricing team, and at Facebook, where I did some machine learning on the misinformation team. After two years of working at companies, I realized that I could have a much larger impact working on something of my own.

6:47And I kept building different ideas that I wanted for myself on the side, starting with an app for podcasts all the way to Julius. Julius is about eight months old, and we didn't expect it to resonate so strongly with our users. You asked me how we came up with the name Julius. So I'm obsessed with history, particularly Roman history. And when I started the company, I incorporated the company as Caesar Labs and kept playing on the Roman theme until we landed on the idea for Julius. And it was pretty obvious to just go with Julius Caesar. Okay. Well, yeah. Where did you learn to code? Was that in high school or even earlier?

7:35and you did a BS in computer science. Did you go on to do any graduate study? Yeah, I did a BS in computer science. I learned to code mostly in my free time in my senior year of high school because that's around the time I watched this movie Social Network. It's on the founding of Facebook. And to me, that seemed really cool that a bunch of 19-year-old kids can build a piece of software that is used by billions of people. So I started learning how to code mostly through the internet. And initially, I was not learning the right things because I was trying to do a textbook way of learning code where I pick up all the concepts and try to learn all of them.

8:31It wasn't until college, freshman year of college, when I got introduced to the hackathon community. And I realized it's much easier to learn hard things or things you don't know if you pick up a project you want to do with that technology. So, oh, I want to learn front end. Oh, let me make a website that does this cool thing for me. And in the process of learning to make that website, you start learning things around front-end technologies, how to write code, how to deploy it, where do you even host a website? How do you even get a domain name? What is a domain name? So I sort of learned most of it through projects.

9:14Yeah, that's fascinating. And then you said that you were building apps. Are any of them still around? Unfortunately not. The only app that's around today is Julius AI. But in COVID, I was pretty into podcasts. And still I am. Thanks for having me today. But one of the things I realized about podcasts was the podcast is a really good piece of information and content. And then when you go into the comments on YouTube, there's additional discussions happening where people share interesting facts and interesting tidbits. tidbits. That didn't exist for other platforms outside of YouTube. So my first ever idea was, hey, can we build a comment section for podcasts?

10:01And it was called podcastcomments.com. Oh, that's interesting. Okay, so let's talk about Julius. This started as a project or was it a business right off the bat? Can you talk about how you came around to that? Totally. So me and the Julius team, we are really excited about the potential of AIs writing code. In fact, Jensen talked about this in his GTC just now, about how AIs will write pretty much all of our code in the future. But we had that insight, you know, two years ago, and we kept trying to build some product or solution that could enable people to do that. Like, what if we take the definition of a programmer and completely expand it?

10:50A programmer is not somebody who is just in an IDE, writing a bunch of lines of code and solving, you know, these system problems. It's a biologist who has a ton of data on their hands and wants to do computations. It's an accountant who is trying to crunch a bunch of numbers using code. and the definition we thought of a programmer is going to expand a lot. And we had that insight. It took us a long time to talk to users and iterate and arrive at something that actually worked and resonated with people. So we started with this concept of, can we get these models to write all of our SQL for us?

11:36You know, SQL is this thing that a lot of business people need to use. And we attempted to do that by creating massive data sets and databases and getting a natural language query engine on top of it to work. And soon we realized two things. One is when people have SQL queries, they don't really change them. You know, once you have a query, it kind of works. Second thing is that these models are a lot better at Python than they are at SQL. And with Python, you get this whole ecosystem of modules and packages that a programmer can leverage to do machine learning, to do statistics, to build applications.

12:23All of that ecosystem is built for you. So after a bunch of talking to users and shipping different ideas, we arrived at Julius. And the insight there was, you know, Python is really powerful. And if we can create these code sandboxes and take the best LLMs and put them on top of those code sandboxes, in a way, give an AI its own computer and let it write code, run code to do different things for you. That certainly is really powerful. So we built and launched that, and we called it Julius. And within the first or second day, we knew this is something people really wanted, and it solved the real problem.

13:03Yeah. First of all, how did people become aware of it? I mean, how do you market it to users or get beta testers and that sort of thing? And two, this is a problem that everyone's been chasing. I mean, DeepMind with AlphaCode and certainly Meta, Amazon with CodeWhisperer, GitHub Copilot. And the problem has been that while you can do code completion for a coder to review and accept, these systems hallucinate too much, even in very minor ways that make it impossible to code complete programs beyond very simple programs or as with alpha code with very strict parameters. So how did you guys deal with that?

13:58Totally. So your first question was, how did we... How did you get the beta testers? Or you say that you had a big response, but there's a lot of stuff out there to get people to look at. Totally. There's a lot of noise out there. and especially I feel like in a way benchmarks are being being gamed today for marketing reasons which is quite quite disappointing because it takes away the hard work a lot of really good researchers are doing but in terms of how we got our early users we had this strategy of using chat gpt's plugin store to validate the ideas that we're building so we built a variety of plugins in the ChatGPT plugin store.

14:45Everything from an AI software engineer to an AI code reviewer, to an AI data analyst and an AI data scientist. And we would build these plugins, we would put it out there, and we would see how do people use these plugins. And honestly, most of the plugins got no usage because it wasn't really a problem people were looking to solve. So the AI software engineer plugin, for example, People would install the plugin and they would ask it things like, make me an iPhone app or make me a website for a recipe. And it would attempt to do that, but the models aren't really there yet. And making an app isn't really a problem that people are looking to solve immediately.

15:34But what was really clear, besides the plugins that got no usage or people didn't retain at all, that the data scientist and the data analyst plugins was something that people were using pretty heavily. And we had very generic names for these, like, you know, AI data analyst and data scientist. and within the week, first or second week, we realized, okay, so the market or the use case where there is a fit right now and what the models can do right now, it seems to be the data analysis and data science aspect. So let's go actually build a better experience for our users. Let's understand what are they trying to do and can we build the best product out there for that use case?

16:21So we basically got beta users from there. How do we go from there to half a million? Well, for some time, you know, the initial users came from the ChatGPT plugin store, and that helped us validate the idea of what we were doing. And after that, what has worked pretty heavily for us is building a community of users online. So on Twitter and on LinkedIn, and then constantly sharing our progress and all the features that we're adding and all the fixes we're making to the core product. So constantly sharing that pretty much every week, sometimes multiple times a week, inspires our users to use Julius more.

17:06It reminds them, hey, Julius exists. And this is a live product. This is only going to get better from here. So I should give it a try. and it helped us from this community of users and it worked in a really interesting way. So one of our users, his name is Professor Kerry Beck. He has been a user, I believe, since the first or second week since we launched. And he's a paid user. And I didn't even know he existed. He just knew about Julius and he used Julius. Then come this spring semester, he reaches out to us and says, hey guys, I've been a Julius user for five months now, and I would love to teach a whole class with it.

17:49So he's teaching a whole class at Rice University. It's called AI-powered financial analysis. And we actually flew to Rice to sit in the class. And it was really surreal seeing an entire class being taught by a Julius user to a whole class of Julius users. So we, you know, So sort of using those early beta users to validate the solution and the problem, then engaging with our users and using social media to spread the word. And then the third step was the word of mouth that came out of it. Yeah, that's fascinating. So on the question of how do you solve the code generation problem, and then we'll talk about how the code generation fits into the product, but how did you solve that code generation problem or is it solved?

18:43Totally. It is solvable today for a subset of the problem. So if you define a subset of the whole problem, which is, you know, AIs will write all of our code in the future and do all of our coding, which is true. and that future is going to come. But where the models are good today, it's on very narrow areas. So what we do is we focus on building the right experience and the product for those use cases. So I just gave you an example of this class of business students learning AI-powered financial analysis. And they can do financial analysis on Julius, upload their datasets and their Excel files and have the AI write code to do that.

19:32There's some other cases where a professor at Yale School of Medicine is using Julius for biostatistics and doing his research. There's a CEO of a hot tub company in the Midwest. He uses Julius to manage his sales data and order data. And all these use cases are related to data and data analysis. And that's what we are focused on. And because we define the focus area, it allows us to iron out all the wrinkles and all the hiccups and obsessively focused on, okay, where did the code go wrong this time? And how can we prevent this from happening again? A combination of that, then using several different models for their strengths.

20:22So we heavily use GPT-4. We also use models like Claude, Command-R from Cohere, Gemini from Google. And right now we are in the process of rolling out Llama 3 for a variety of use cases. And the way we do this is we take one model, you get it to write the first piece of code, and then run that code in a VM in the cloud. So the code gets executed. Now, sometimes that code doesn't compile because it has errors. And when that happens, we, in a loop, try to fix that error using either the same model or a different model. We use a mixture of experts approach where we consult different models, give it the code, give it the error, and say, hey, what went wrong here?

21:17What do you think the error is and what the fix is in this code? And what that does is it increases variance. It's kind of like asking your friend or your colleague to proofread your work. They're able to find issues in your work that you usually look over and look past. And it's sort of like using a variety of different models to review the code, fix it, and then run it again. And then you can sort of do it in a loop until the code compiles, and it actually gives you what you want. There is a trade-off between speed and how long you want to do that. And we always play with that sliding window of how often or how long can we retry before the user gets frustrated.

22:04But usually it's within a minute or two the user gets to see an output. Yeah. Just on that latency, do you have a sense of how long does that process take, that looping to refine or fix bugs in the code in the background? Because from the user's point of view, they, well, maybe we should start or go back and explain what Julius does. It's an interface for tabular data, right? For analyzing tabular data. Is that right? Yes. It's an AI data analyst or AI data scientist. And so give us a use case. Is it a plugin or do you have your own spreadsheet in the product? Yeah, great question. So if you have an Excel file or a CSV file, sometimes users even upload images and PDF data.

23:01You can upload any data to Julius and have the AI analyze it for you. It's better if the data is in a tabular format, but we are working pretty actively on making unstructured data also queryable and making a lot of progress on that. What's happening under the hood is when you give it a data, think of it as your data analyst. You would give your data to a data analyst, and the data analyst would download the file, go to their computer, write some code, write some scripts, either it's Python or SQL or any programming language that they're good at, and query the questions you ask it. So let's say you give it data of a scientific experiment.

23:49a million rows of data, and you say, can you perform regression on this and find me, what are the trends in this data? This is a pretty open-ended question, but the AI will basically take your million rounds of data, write code to first take it apart, understand what's in the data, where are the different outliers, where are the different columns, what is the unique values in each column and then proceed to write a regression on top of the data. Then run that regression again to give you the output you asked for. Usually to a human, this would take a little longer than what it would take Julius today.

24:34And Julius is always available. And it's nearly instant compared to most humans. and what's really useful to a lot of our users is they're not coders, they're scientists, they are financial analysts, they are business analysts and their expertise is their domain and they're able to leverage Julius as the coder to perform all this statistics, all this data science and data analysis for them as long as they ask simple questions. Yeah. Yeah. And so when they ask a question in natural language, that question is sent to GPT-4 for parsing into a series of steps. Is that right? and then each GPT-4 writes code for each step and then that code is executed and if it doesn't compile, then what happens there?

25:53It's sent to LAMA3 or sent to our commander or whatever. Yeah. Totally. So we have a big meta planning step where if you ask for tasks like regression or ANOVA analysis, that task requires a series of steps. You probably want to first load the data into memory on that computer. Then you want to break it down, understand what's in the data, do some data cleaning, find some outliers, find some unique values. and then you dive into your actual ANOVA analysis or regression analysis or whatever questions you have about your data. So we have this meta step which comes up with a series of steps in the plan.

26:44And then usually that model is GPT-4 and then each step correlates to some code and the code gets written, executed and ideally the code compiles without any errors. But what a lot of our users who use tools like ChatGP or Microsoft Excel Copilot, those people come to us and tell us those things barely work. They're riddled with errors. And what we're focused on, as you put it, is code generation is really hard to get right. And we are able to identify and collect metrics on what are the most common errors and how do we prevent those errors upfront? Either a combination of prompting or through rule-based engines or a third step, which is using a mixture of experts where you run that code, the road compiles or doesn't compile.

27:40If you have an error, you pass it through a variety of different models and that stack gets changed often. So I can't really share more on what the stack is today because by the time the viewers, let's say a year from now, It could be a completely different stack. But we give it to a variety of different models. And we say, hey, here's the code and here's the error. Can you fix the errors in this code? What are the things we can do to recover from this error? And sometimes the AI is able to recover from it instantly. Sometimes it can't because of a dependency issue, because of data issues, there's a variety of issues.

28:22and if there are certain issues that are on the user's end, how do we build the experience to surface that back to the user? We tell the user, hey, you want to find your highest paying customers, but there is no customer column in the data. Are you sure you want to maybe add that column when you export from your CRM? And how do we surface that back to the user is another focus area for us. right and then uh how often does the code just never compile that or or that the loop becomes because when auto gpt first came out you know on github i played around with it and you know i'm not a coder so when it would hit an error i'd put the error in chat gpt or gpt for and it would give me a fix and I'd put it back into the virtual studio and then it would come up with another error.

29:29And I realized what it was doing is the fixes, you were just going down like a rabbit hole of one error compounding another error and pretty soon you're way off in the weeds. And so how often does it not compile ever? Or how often does it, you know, what's the longest that it will take if it's fixable to finally compile? Are you talking about 10 minutes or an hour? Yeah. Totally. So that is something that depends heavily on the task. One of the things is sometimes our users paste a link to Amazon and say, can you go scrape all products from Amazon? And what Amazon does is Amazon will pretty heavily prevent bots and scraping.

30:30That's a huge part of their business. And even if the AI writes code to do that and it will fail to scrape all products from Amazon, it will get an error and it will try to recover from that error. And that's a kind of error which is sort of nearly impossible to recover from. But when we focus heavily on the use case of data and data science and data analysis, we're able to squash out most of the common errors, actually. We have made so much progress on that front from collecting metrics on where are the common errors, Why do they even emerge? How do we prevent them upfront using rules or a mixture of experts?

31:14And squash those errors out. In terms of how often does it happen? About 70 % of the code that the AI writes compiles cleanly as of today in the first try. That means 30 % of the times there is a compilation error in the first time, which is pretty low compared to most so-called agents out there because we're focused on narrowly on solving this real problem that a lot of people have. In most cases, the AI will continue to retry for up to 10 minutes and then surface, but if it doesn't succeed in those rare cases, we surface that back to the user so the user can make a decision on, hey, like it's not possible to scrape Amazon, but it is possible to do financial forecasting even joyous.

32:05Yeah. And then, well, that's interesting about scraping. So this does more than just analyze data in a spreadsheet. What's the range of tools, of tasks that it can cover? Totally. We would love to support anything that's possible with code. And I think that's the future. The future is there will be 7 billion programmers and they will all program with English. Whether it be scraping, whether it be making software, whether it be making apps, websites, or doing data science and statistics. All of us will be programmers because we will be programming with English. Today, we're narrowly focused because that's what solves a user problem immediately.

32:56And we have, just in a short amount of time, half a million users who are looking to solve that problem with us. as the model capabilities advance more and more use cases will open up so scraping is something that sort of works today and you know one of our users he is he he uses julius to collect data from the internet for his own research he's a professor and he and his lab use julius to collect data from from sites that don't prevent scraping. They're open to scraping and collecting data. He uses Julius to automate that data collecting process and collect all the data, structure it, and analyze it.