Rajarshi Gupta: Artificial Intelligence and Crypto at Coinbase

0:05Hey, everyone, and welcome to Generative Now. I am Michael Magnano and I am a partner in Lightspeed This week on the podcast, I'm sharing a conversation with Rajarshi Gupta, the head of machine learning at Coinbase. He joined us on the Generative San Francisco stage last year with my colleague Anand Iyer, a Lightspeed venture partner focused on crypto. This is a really interesting conversation about how Coinbase is incorporating machine learning into every part of the company. Rajarshi also answers audience questions and offers some great advice for founders. Enjoy. So welcome to Generative SF.

0:40My name is Anand. I work on crypto here at Lightspeed. And I'm excited to have Rajashi here. I'll have him introduce himself. And just as a reminder, this is a session that's focused on meme coin tradings. Rajashi is an expert. I'm completely joking. The AI folks here are genuinely freaked out. So I'm going to retract that statement right away. But why don't I turn it over to you, Rajashi? Maybe you can give a little background on yourself and everything from Genesis Block to how we got here today. Yeah, sure. All right. So I think going forward, I did my PhD across the bay at Berkeley, and then I spent 10 years at Qualcomm Research.

1:13And at Qualcomm Research, I guess my career, the funnest thing I did was that we built the industry's first on-device machine learning engine. So this was way back and we launched it in 2015. It was the first on-device machine learning engine meant to catch malware on Android devices by looking at everything that was going on on the phone. And so, you know, I was really excited about it. This was my zero to one project. It was my idea. I was the only engineer on the project at the beginning. And then we kind of built it and shipped it and so on. So then after my Qualcomm journey, I went and did a couple of startups.

1:46Both startups were in security. The first was a small startup called Balbix, which is a Mayfield startup down in San Jose. And then I joined a much bigger startup called Avast, which was actually the world's largest consumer security startup. There were 500 million active users. And then we went IPO. And then since then, we merged with Norton for like$8 billion. And now the new company is called Gen. Then I joined AWS, hadn't worked at a very big company, joined AWS as a GM in the SageMaker team for a few years. And now for the last three, I've been leading AI at Coinbase. No, that's perfect. Thank you for being here.

2:25I don't think we talked about this Android experience. I'm curious because that was pre-generative AI. Oh, totally. And so, and then you had to, I mean, malware and Android has been a thing forever. So yeah, maybe like, I'm just curious, maybe let's unpack that because I don't think we've talked about this before. So like, what was that like? Yeah. And it's like, was this model happening on device? Oh yeah. The model was happening on device. In fact, the entire model was written in C and we had to write our own training algorithms because there was no tools at the time. And we wrote our own training algorithms.

2:54The model was written in C and it had to be written in C because you couldn't put the model on the regular Linux stack. It had to put it in the internal secure stack of the phone so that the model itself could not be hacked, the code. So that was a really funny, we were learning how to do training. Nobody really was doing it well at the time, right? This is like 2011, 12, 13, right? We had to do how to do training, we had to write the code, we had to write the training. Well, the training was happening offline, but the inference code had to be written in a very optimized way. It was looking at everything happening in the Linux layer all the way to the app layer on the Android devices.

3:31And it could literally catch a ton of malware. So we shipped from between 2050. I don't know, most people here probably have iPhones. But if you had Android, so we shipped it from 2015 to 2019. It shipped in all the high-end Huawei and Samsung and LG phones. So eventually it shipped over a billion chips. That was a really fun experience that we had. What was the learning process like? Because how do you know when you were successful and how did that kind of feedback loop go back into the models? So when we first started doing it, you know, I remember, so Qualcomm is a nice company that, you know, as I think at the time I was like a staff engineer.

4:07I mean, as a staff engineer, you have a cool idea and you could literally go up to the CTO and present the idea. He was coming to our office and I got a slot to present the idea to him. And Matt Grob, and he basically looked at the idea and said, this is a crazy idea. It's never going to work. But if you think this is going to work, I'm happy to let you try. So go ahead and try it. I mean, this is what we are supposed to be doing, right? You know, I won't be doing my job if I didn't encourage you to go try this thing. And so don't believe me, just go and try it. And then I tried it. I was the only person that I got a couple more people and a couple of interns to help me.

4:44And then we built the prototype. And the prototype was we're taking, there used to be a version of Android called, I'm forgetting the name. So basically you could take that version of Android and make changes to it on your own. And then we made it work on a real phone. There was real malware and it was actually stopping and catching it. And then basically the power of the demo. The demo was, hey, here's the real phone. This is a real malware. We had two phones. One was not catching it. One was catching it. And then people said, heck, this works. So let's try and build it together. And we did. Amazing.

5:17Obviously it's been a while since then. And you're now at Coinbase working on machine learning. And yeah, it'd be great for everyone here to understand what does your role entail? What do you do there? And I know you have a fairly large-sized team as well that's pretty geographically dispersed. So the first thing we do, which is probably the most well-known is, of course, my team makes sure that every transaction, every login is protected. Like, as you might imagine, it's not just Coinbase. If you have a PayPal account, if you have a Visa account, every transaction is protected by a series of machine learning models.

5:48And in crypto in particular, there's a lot of people trying to attack your account and we protect it. Literally, every login, every transaction. So that's one piece of it. The second fun way of thinking about it is, you know, you open your app, we get to decide what you see. Honestly, I mean, you know, there's so many apps, so many assets, so many things. At every stage, you do a transaction, you see a bunch of other assets that you could buy, or you get notifications. Naturally, this like Facebook or Instagram, right? Or LinkedIn. The notifications we send are targeted towards you. We decide what you get to see.

6:21So these are very traditional. Like one is traditional finance applications. One is traditional web two applications. With the app, we have to do these things. I have so many questions for you. Maybe you'll start with, because you've been there for three years and I feel like generative AI. Oh, sorry. Yeah. I mean, I knew you were going to ask. I just told you about the rest of it. And then I have a whole big gen AI team. But let's talk. Let's get there. Yeah. I'm curious, like what, Amis, maybe if you could share a bit about as much as you can about the kind of models you've developed. How much of it is off the shelf?

6:48Are you using any known flagship foundation models or so on and so forth? I mean, one of the things we did was that when the generative AI explosion started, I would say early, like end of 22, well, basically 3.5 came out, which is Thanksgiving 22, right? So very soon after that, we made a strategic bet that this is going to be a game changer and we are going to invest heavily into it. Now, this was not a very good time for Coinbase because this was at the bottom of the crypto doldrums and we had just had a big layoff and stuff. But I was like, all right, this is the technology. This is what is going to really get us, help us.

7:27So we are going to focus. I mean, okay, we're not hiring, but we're going to focus everything we can to do this. And I can tell you that we released our employee assistant in fall of 23. So it was like pretty early. So all company employee assistant was released just after Thanksgiving 23, actually. And the employee assistant is, what does that do? Okay, so I'll tell you. So what are the things we are doing with Jai? So there's two halves to the projects. So one half is to essentially build something for all our employees. And the other side is to build an assistant for all our customers. Now, naturally, the employee side is easier because you're talking to your own employees.

8:08You don't have to worry about the thing. and so on. The first big release was there. We're doing two things. So first one is very straightforward. We have an employee assistant persona where every one of them has an integration with Glean to be able to get access to your data so that it can give you answers based on what you know. Enormously useful. A lot of people use it. Then there's a whole series of other things that my team initially built, which was to help different people, like our designers, things that help our finance people, things that... So we built several of these ourselves. One of the most popular in the company is that we just went through our performance review time.

8:52So we have a performance review assistant, which you basically take your notes of your bullet points and you drop it and it writes your performance self-review for you in the Coinbase format with the correct structure, with the thing, with the correct word limit, and then you need to edit it and do it. So this time we had out of, I think, 3 ,200 people, 2 ,000 some people used it. So everybody across the company, all the way to the C-suite, uses it. We can't see what they're doing with it, but we can see that a number of people in our exec team are using it to do this. So that's one example. So then what we did is that we really wanted to build it as a platform.

9:33on. So we released much of the functionality as an API such that other people could build on it. So like not the entire company doesn't have AI expertise, but we built it such that other people can use it as an API. So like somebody went ahead and built a really nice one, which we built what is called an incident bot, which is that when there's an incident going on, you get into the Slack channel. And basically, if you look at the history of Slack channel, it's basically somebody comes in, asks what the heck's going on. Somebody answers. 10 minutes later, somebody else comes in and ask what the heck's going on.

10:02And then things have changed and stuff. So now every person comes in and you just ask the bot, saying, hey, what's going on? I'll DM you back exactly what's going on. And this was built by that team. We have our data science team who built a text to SQL bot for it. And then, you know, there's all these things. So that's the other one. But now this is for internal employees, right? Now for external customers, we released the first few releases where, of course, you know, We are planning to do an assistant for all your user journeys. The first user journey that we tackled is the obvious one, which is the support user journey.

10:38So we released our LLM-based chatbot in November. So we're now handling whatever, like tens of millions of user requests. With that, we took over the search on our site. So now if you go to do a search on either on your phone or on the website, you'll get a Gemini-like answer first from AI. And now we are doing other things like, and this hasn't released yet, but we are working on things where when a customer tries to do some research, customer is trying to do, when we provide insights on things, customer is trying to look and find out about certain types of crypto assets, which is pretty complex.

11:15We will help you. And then this is where we are going to expand. Got it. How do you evaluate and manage deployments of these kinds of experiences, assistants, agents, models? Like what does that process look like? So the way we do that is that, so this is called CBGPT, Coinbase GPT, and this is a platform. So it's truly a multi-cloud, multi-LLM platform. So we literally use models from Azure, GCP, and Amazon. You asked the question earlier. So the two biggest use cases, which is the chatbot and the help, are both on cloud, but it's load shared between AWS and GCP. Now, the way we do it is that we have, this is one of my biggest problems with the whole LLM space, right?

11:56So I grew up with machine learning. I've been working on it for, I don't know, now 15 years on machine learning. And then before that, I worked on statistics. Entire life, we've dealt with, here's the prediction, here's the confidence interval. All of a sudden, you're in the space where here's the prediction, and I don't know the confidence interval. So it's a very uncomfortable situation, right? That for LLMs, you don't know how good the answer is. So you're having to do all these weird things as a different LLM, as a judge, and all these things. And it's a constantly moving space, right? Like you had an LLM, you're using somebody else as a judge.

12:32Now this LLM is better than the judge all of a sudden. So now you need a better judge. So this gets very hard. I don't have a solution. We're doing same thing as everybody else is doing. We have an evaluation portal where people can try their own ground truth sets, which suck because people just don't have a sense of, like, you're a normal user, right, of the app. How do you know what's a good ground truth set is to figure out whether you're doing this. And then we are doing LLM as a judge. And then we are doing human evaluation and we're doing curated data sets. Nothing fancy. It's just what everybody else is doing because I haven't seen any good answer.

13:08So lots of startups, you know, evaluation of LLMs, truly great, great, great problem to solve. We've gotten plugs for a couple of Lightspeed portfolio companies already. We've got Glean, got Anthropic. Okay. We'll keep it going. Patronus, we'll keep it going for evaluations. But that's really helpful. By the way, for folks who have questions, it's a very small, intimate crowd. So if you have anything that's coming up that's in the moment, feel free to fire away. But I'll park that for now. I'm on Anthropics Customer Advisory Board and they've been a really good partner. So for our chatbot, right?

13:40When we were releasing our chatbot, one of the biggest scared thing was that we were releasing it in June of 2024. If you just think back, not that many companies, like Uber didn't have their chatbot release at the time. So we were like, wow, we're really pushing the edge and we were scared. And naturally, the fear was about the guardrails, right? What if you get your chatbot? What if somebody, I mean, the New York Times front page scared, right? And they were very nice and they proposed to do a joint venture with us. So we built the guardrails model, a separate guardrail model in collaboration.

14:12And they were super helpful because we are, I mean, we're one of their early customers, big customers. And we did a really nice thing and it was, it kind of saved us so much time. Yeah. What's keeping you up at night these days? I think, I mean, honestly speaking, you know, I tell people that most of your life at work, the general feeling is that you're pushing a boulder uphill and there's gravity. And it's pushing it down and you're fighting something, right? Like once in a time in your life, the boulder is rolling downhill and you're chasing after it. So right now we are in that phase. And I have been since for the last two years.

14:50So it's awesome. My fear is, you know, I've been doing this head of AI kind of role for a while. And most of the time you have to tell people and people are used to the way they do things, right? You have to go and convince people that, hey, this way is better. You're going to do this and stuff and so on. And now all of a sudden the whole thing switches and people just come and say, can you please do this for me? And so on. So I think that's the thing. So the part that keeps me up at night is the fact that our people don't plan for the amount of effort and the amount of sophistication it takes to make these solutions real.

15:25So I'm not talking about the hype bubble or anything, but it's just that people are people's expectation. We can match the expectation, but that doesn't come for free. Sure. I mean, I've started telling people that, you know, AI is like magic. But before you can do magic, you have to go to Hogwarts for seven years. So it's that seven years, which is hard. And people don't get the fact that you can't just put an LLM and it'll do the work. You have to do a lot of work in getting the plumbing right, doing the testing, doing the measurement, doing the analysis, and, you know, do seven iterations of it.

16:03And then it becomes really good. That's a really interesting point. But do you use generative AI in like, you know, you have a pretty large team. How has, are you using tools like Cursor or V0 or anything like that? Like how is the Eng team starting to adopt? That's a great point. And, you know, that is one of the great cases where we did a straight away buy over build decision. Like as we were beginning to look at generative AI, that was right when, I mean, if you remember, GitHub came up with Copilot. but within like two months of the 3.5 being released, and I just analyzed it with some of my team and said, whoa, this is a good product.

16:40And we adopted it. So right now in our company, we have rolled out for everybody, which we've rolled out Copilot, and then we did Sourcegraph Cody. And just today, we rolled out Cursor to everybody in the company, all engineers. So it's like, we think these are great. You know, they are doing extremely good thing. All the developers love it. and we are adopting it. The funny thing though is that, so you read, and I don't know how well it happens. I'm sure every company is beginning to measure this, right? So you say, okay, 25%, like Sundar Pichai is now, 25 % of code is being written by things.

17:16So if you're the CEO, you think 25 % of code is being written by AI. Awesome. I can have 25 % fewer developers. Or we're going to have 25 % more time in there. But it turns out that developers don't code for eight hours a day. They only code for like two hours a day. The rest of the time they're going to find, they're trying to find data, find what happens, do a debugging and so on. So the total, even if 25 % of code, it's like 25 % of two hours. So you're saving like half an hour a day. I think the bigger advantage is coming from these, is going to come from these systems or these agents, which not just can like predict the next three lines of code, but actually understands the problem.

17:59It's a much harder problem. I mean, software, if you think about it, most of us have been software developers in our life. You don't just sit and write code all day. You spend a lot more time figuring out how to solve the problem. And then the actual coding of it doesn't take that long. Yeah. I was talking to a friend of mine who works on Gemini, and he was saying that 20%, 25 % stat was literally about like auto-completion of code. Yes. Not about, you know, writing code itself. So that's kind of a misleading statistic. You know, when we talk to some folks about some of the issues they're facing when it comes to the adoption of AI, usually like GPU shortage comes up, quality assurance comes up.

18:37You know, is that something that's been on your mind too? Oh, so like I was giving a talk at GCP, the Google Cloud Next a few months ago. And one of the things they gave me was like prompts they gave me was like, what keeps you awake at night? and I wrote down in my slide, you guys not giving me enough GPUs is what keeps me up and down. And then, you know, the person who was looking at it and she was like, I need to get that reviewed by somebody. But thankfully they didn't. I mean, they were completely fine with it. Like whoever reviewed it said, no, this is the right problem. So, you know, to my big surprise, getting the available GPUs was the biggest problem we faced.

19:18Like by far, like entire last year, that was by far the one that caused me the most grief. And it's because, so, I mean, at the beginning, we did our employee assistant and these assistive things for our agents and our developers. Great. You know, whatever. 8 ,000 people are, like, 600 people are using it. You don't hit any bandwidth, right? Then you suddenly switch from 6 ,000 people to 6 million people. And then you realize that this burst are coming and there just aren't any GPUs. So we had a couple of instances early in the year when it went down and we really struggled with it and literally had to go and escalate it to both GCP.

19:58That was actually the reason why our main solutions are across both AWS and GCP. Literally, that's the only reason. There's no reason to do it otherwise. It's the same model. There's load balancing and the ability to get capacity throughput in those places. And I don't blame them. I mean, I used to be a GM at AWS. I know Atul, who is the GM for Bedrock at AWS very well. Their problem is that these new models are showing up every month. And when a new model comes, you don't know whether to run that model on 1 ,000 GPUs or 10 ,000 GPUs and how the demand across models are going to switch around and which ones are going to go to LAMA and so on.

20:38So there's not enough predictive ability on it. And just to unpack that a little bit more, can you tell us about what is the workflow? So because you're using specific hosted instances and putting these models, are they like specific weights or specific kinds of models that you're hosting on these GPUs? We have many different use cases. For some of our use cases, we are having our own trained models based on the Lama family. Got it. These are hosted internally, but these are not high bandwidth ones. These are typically the ones where we have some legal security reasons that we don't want the data to go out.

21:14But these are small use cases. No problems, right? The big use cases are hitting, for us, there may be other companies who are doing differently. But for us, the big use cases are hitting these models, particularly, actually, both Claude and now we have Gemini also. Both Claude and Gemini. Now, what happens is that even if you think of it as something as simple as a chatbot, which is the most common use case, right? So it's not like the user says something and we send it to a chatbot. It's a chain with between five to nine LLM calls within that chain because we have to figure out, is the user saying something bad?

21:53What does the user mean? Because somebody, they says, can't send crypto. It's one of the most common things. All right, you need to get more information. You need to get context. You have to figure, do a rag call. You have to get information. Then you make a call. Then you have to change the system so that it sounds empathetic, depending on what the answer is. So it's actually, depending on which side of the flow chart you're on, it takes between five to nine calls. So that's what we're doing. And when we hit these many, like every customer, million customers, you take these calls, you hit them with nine calls, it adds up pretty quickly.

22:27Is there a requirement for specific kinds of GPUs? Like, is there homogeneity that's needed for the kinds of models? Or like, is there a specific kind of instance that you always need from GCP or AWS? I mean, we try to get the biggest one we can get. So it's not, I don't think, I mean, you know, we don't really specify because we are not doing it. I mean, we're not big enough to demand an isolated cloud instance. That's too big. So we are shared instance. So we basically work off latency. and bandwidth. I see. Got it. You know, if you had to look out maybe a year from now, you know, as you're starting to build up your expertise and obviously Coinbase is on a roll, crypto is doing well, so there's a lot more, there's added pressure on your team.

23:10I guess, what is your team going to be doing a year from now? What is, you know, what are the deliverables that you want to hit over the course of the next year? I think there's two axes where we're really trying to make and the two axes are at conflict with each other, right? So one axis is that we always worry about the bull run. So, you know, a bull run is happening right now. We don't worry about a bull run. Just to be clear, the opposite problem. We worry about a bull run all the time. I mean, in fact, honestly, but I don't know if you know there, there's this for those of you who are in crypto, the forever, the meme has been crypto goes up, Coinbase goes down because the Coinbase site, whenever we, this is the first time, like literally we are super proud of it, that the election time and all the crypto run, our site stayed completely up with no problems.

23:53And because of the enormous amount of investments we've had to do in the platform to make it all work smoothly. But of course, there is a capacity issue, right? I mean, we are handling what's going on right now just fine. But what if it goes up 10x? And so that's one side of it, right? So it's not just me. Everybody in the platform is worried about the fact that we don't want that to happen. And there's a lot of work that goes in. And we have some headroom, but I don't know how much. I mean, we're estimating how much. But these things are very hard to estimate because when that many users come, there's so many we do load tests and do everything.