Eradicating Machine Learning Pain Points with Weights & Biases CEO Lukas Biewald

0:05We've talked to many practitioners who are pushing the state of the art. This week on the podcast, we're exploring the dominant ML developer tool, Weights and Biases. Alad and I are sitting down with CEO and co-founder, Lucas Biewald. He has a knack for creating companies that support pain points in ML development. His first company, Figure 8, addressed the problem of data collection for model training. And his second company, Weights and Biases, has created an experimentation platform that supports AI practitioners at companies including NVIDIA, OpenAI, Microsoft, and many more. Lucas, thanks for doing this.

0:35Welcome to No Priors. Thank you. Great to be here. Lucas, you studied at Stanford, where I assume you discovered your interest in machine learning. And under one of our previous No Priors guests, Daphne Kohler. Can you talk about when you started working in AI and learning from Daphne? Yeah, totally. As a kid, I was obsessed with playing games. And I got really into Go. And I was super into the idea of, or thinking about how would computers win at these games. And so I actually sent Daphne an email, maybe as a freshman, being like, Hey, can I work with you? I'm really interested in games. I want to learn how to beat Go.

1:11And Daphne wrote me actually a pretty polite email being like, that's not what I do. Go away. A few years later, I took her course. And I studied math at Stanford. And I have to say, Daphne cared about a thousand times more about teaching than even the best professor in the math department. And so it was really just eye-opening. I just loved how much she actually cared about teaching. And it got me really excited about the AI that was working there. and I went on to be a research assistant for her. And the funny thing at that time was like nothing really worked. Like it was just before kind of, you know, Google was thought to be really like page rank at the time was the thing that was making them work.

1:48And I think later, you know, it became clear that machine learning was a big part of that. But really when I was doing ML, it was like searching for applications that were working. And Daphne was actually really obsessed at the time with a thing called Bayes Nets, which you don't hear about too much anymore because I don't think they ever really worked for many applications. I hope I'm not offending anyone, but that's my understanding. I actually think the thing that I really took away from Daphne that really lasted with me was, I mean, she's one of the smartest people I've ever encountered. And she had this incredible clarity of thought and an intolerance for sloppy thinking that just really served me well.

2:28And I think that's sort of separate from machine learning. You'd see other professors would come and give guest talks and, you know, they would say something's kind of lazy and like we'd all just be sitting there just like waiting for Dafty to like eviscerate them. And I think her personality has mellowed a little bit over time, but I kind of miss, I just miss that sort of like aggressive, clear thinking. And I really admire it. I don't think we got a taste of that, but we did talk about whether or not probabilistic graphs are coming back a little bit. How did you go from, you know, Stanford to founding Figure Eight?

3:03Yeah, you know, it's funny. I actually really struggled doing research with Daphne. Basically, the things that I tried just barely, barely worked. Like, you know, I published a couple of papers that I feel kind of ashamed of where it was sort of like, go from like 68 % accuracy to 70 % accuracy in a task nobody cares about by throwing like 1000x to compute. And by the way, kind of guessing the most likely answer is probably like 64 % accuracy. So it felt honestly kind of pointless and sad. I love the idea of computers learning to do things, but it's hard to sort of sustain the enthusiasm for that when everything you try just completely doesn't work.

3:45And even the things that do work, you kind of wonder if you're like p-value hacking. Like, okay, I tried a thousand things. You know, so I guess something's going to be like a little bit more accurate than a baseline. What tasks were you working on? Did you end up working on Go or games or anything? No, Daphne is not interested in games, let me tell you. And it's actually another, I kind of admire that perspective too, as much as I love games. I'm a Go nerd, so I'm curious. Oh, you are? Oh, me too. Yeah, I love Go. Yeah, Daphne was very not interested. She really was practical. And so I worked on a task that you really don't do now.

4:19called word sense disambiguation where you're trying to find out like, okay, I have the word plant. Actually, if you look in most corpuses because they're government generated often at the time, plant typically will mean like the power plant sense of plant or cabinet often means the sort of president's cabinet sense of cabinet. And so you're kind of trying to figure out like what is the meaning here of these words and then applied it to translation. It's a cool task. I mean, and actually it turns out, I think that these, again, nobody kill me, But my general sense is that these sort of like linguistic oriented strategies really don't work that well.

4:54It's kind of like by feeding more data in and sort of like working on outcomes, you can figure these things out much better. So a little bit of a dead end. And actually, you know, I was so frustrated by that that I just really wanted to work on something that people cared about. I actually turned down an offer from Google because they didn't tell me what I would be working on to go to Yahoo because they were like, okay, you can work on search ranking in different languages. But that actually turned out to be incredibly fun, right? Because it was super applied. It's actually a task that works really well.

5:27And Yahoo is kind of in the infancy of switching from hand-tuned weights to machine-learned weights. And they really had not many people actually working on deploying this stuff. So I was writing code to translate machine-learning algorithms into C code and then we would check get into our little code base and run this kind of like semi-hand generated C code in production. So that was super fun. But the thing I learned there, actually, which I think I'm not the only one that learned this, but I just felt it. I would go from country to country trying to switch from hand-tuned weights to an ML model.

5:59And I was sort of the messenger here. So sometimes it would work and sometimes it wouldn't. And so people were either really happy with me when it did work, or they'd be really pissed at me when it didn't work. But I kind of realized, actually, the model that I'm building is like the same for each country. It's the training data though is different. So some countries would take the training data collection process really seriously and they'd get a great model. And some would just like really half-ass it or like, you know, have these crazy like issues in the data collection and then the model wouldn't work.

6:28And so I just really kind of viscerally felt how much the training data process mattered. And I kind of felt like, you know, why don't they let me get involved in the training data process, like that would be a better use of my time than building these models. And so I wanted to make a company where the people doing the ML could actually have control over the training data collection process and really get like visibility into it. Because, you know, at the time, I think the thinking was like, oh, this is sort of like a manual task. That's like more of like an operations team should deal with this.

7:02And they would like, they would do this thing where you'd like make this giant requirements document, it was so like waterfall. Like it would be like, yeah, it wasn't iterative. Oh, it wasn't iterative at all. And it'd be like, you'd make like a 50 page document. And like, you know, that the people doing the labeling are not like reading that document, but you kind of need that to like cover your ass if they did like labeled something, you know, not the way you want. And it would have been so much better to be like, look, we're trying to write search results. Like put yourself in the mindset of like someone, you know, who's like looking at this, like, is it good or bad versus trying to lay out in like excruciating detail what makes something relevant or not relevant.

7:36I think also at this time, like when you first started, I think originally it was called Dolores Labs and then Crowdflower and then eventually figure eight. Like I think I met you in your Dolores Labs days or something. I know, I remember, yeah. Yeah, yeah. And at the time there weren't really solutions for data labeling externally, right? Some people are using Mechanical Turk from Amazon to sort of run jobs on untrained workers. There wasn't like scale, there was none of these services. Yeah. And so you got really early to this idea of starting a data labeling company and that that was actually very useful for machine learning.

8:05And so it'd be great to hear what were the early days of that like and what was the industry like and how did you get all that running? Yeah, I mean, it was funny, right? Because back then I was coached actually quite a lot by Travis Kalanick, who's famous now for doing Uber and other things. But he was like, don't tell anyone that it's like AI, like VCs don't want to hear AI, which actually good advice at the time. And it was good advice in the early days of the company. And - Sorry to interrupt. I think one interesting side note on that, just from a Silicon Valley history perspective is Travis used to have these effectively like hackathons or meetups at his house called the hackpad.

8:40And, you know, I think you used to go to those, you know, a bunch of friends of mine used to. And so a lot of startups actually had some impact or influence from Travis in those days, like due to his fact of like, you know, being another founder in the scene and kind of getting everybody together. And so it's kind of an interesting moment in time or in history. And to your point back then, like AI wasn't really as popular as it became later. So it's kind of an interesting like side note. Well, I mean, not only was AI not popular, but like startups weren't popular, right? Like my family didn't, you know, understand about startups.

9:11And I had graduated Stanford. You'd think I'd have all these great like connections, but it didn't feel like that. Like I had no one who knew how to like raise money from VCs. I didn't know any, you know, VCs or I didn't really know any like entrepreneurs, honestly. and we had this website for Dolores Labs in the early days, just trying to get customers. And it put my personal phone number. I actually remember I was like the first user of Twilio because I needed to make a phone tree. And so I used Twilio software. And then like all three of the founders came to my house to like help me like make that phone tree like work better, which is kind of amazing.

9:43It was like, you know, like, you know, one of those like, you know, 20 something, like, you know, grungy apartments in the mission. And then Travis called in, but you know, it's funny because the phone tree, We were just trying to pretend like we were a big company. And Travis called in because of the phone numbers on the website, not because he wanted to buy anything, but he just thought it was awesome. And so I'm just like, I pick up my phone and then there's just this guy in the other and just be like, oh man, this is so cool. I'm like, okay, who are you? It's like, do you want to get coffee?

10:13And that actually turned out to be incredibly helpful. But then I think the thing that was so different back then is that the people doing ML, there just weren't that many like there were people like heavily investing in ml but there but it wasn't that many and so what happened was you know we got like ebay as a customer which has really mattered at the time and we got like you know google as a customer and bloomberg and then there just like wasn't anywhere else to go so like you know my board was always like recommending like read crossing the chasm and and we tried like a million different ways to like you know grow the company and you know i don't know i hope this doesn't sound defensive i mean maybe i was just a bad CEO, but we had years of struggle because there was no chasm to cross.

10:57There was nowhere else to go. So we tried all these different things to build more complete solutions for our customers, and it just didn't work. And then all of a sudden, autonomous vehicles got popular. And that really actually suddenly caused our revenue to start to grow really fast again. But it was like an eight year lull of really no growth. So it's hard because we started off fast, got everyone really excited, kind of got whomped for just years and years and years. Actually, we had all these competitors. They all went away. So at some point, we had no competitors left because everyone had gone out of business.

11:34And then it was a funny experience because Scale came along and totally ate our lunch in the self-driving market, which is a market I knew and loved. And so I was so excited to sell the company after so many years of struggle. But then right after that, we see scale just skyrocketing in revenues. Like, oh man, I wish we had just maybe held on a little bit longer, but then it gave me the space to start Weights and Biases. So who knows? I want to be like Daphne Culler and evaluate my decisions accurately and critically, but it also does seem like I've had some good luck along the way. Yeah, no, the market's shifted so dramatically.

12:10And I think to your point, self-driving was the first time that you suddenly had a bunch of systems at scale that people needed data labeling for. And then of course, now we have this LLM wave, but it's all very, very recent. And I think a lot of people basically view ML as this sort of continuity and everything has always been kind of rising in a sort of almost linear way. And in reality, it's this very bumpy set of discontinuities in terms of the set of technologies and markets that people are adopting it in. And so it's not continuous. It's a discontinuous thing and nobody thinks about it that way.

12:36When you started Weights and Biases, you said something along the lines of, you can't paint well with a crappy paintbrush, you can't write code well in a crappy IDE, and you can't build and deploy great learning models with the tools we have now, I can't think of any more important goal than changing that. And that's, I think, like when you announced that you were starting Waste and Biases. And so I was just curious, what lapses and capability really got you going on 1B? And can you also just, many of our listeners know what it does, but for those who don't, could you explain what the product does and how it works?

13:08Sure. Yeah. So it's kind of constantly evolving, right? Because we're saying it's like a set of tools for people doing machine learning. We're best known for our first thing that does experiment tracking, which keeps track of how your models perform over time as they learn and train. But we also have a lot of stuff around data versioning, data lineage, production monitoring, model registry, the end-to-end stuff that you'd need to do machine learning reliably. And I think the thing that happened to me was I had been running crowdflower for years and I always loved machine learning, but I was like really starting to get out of date.

13:43Like deep learning came along. And at first I was kind of skeptical of it because people are always saying, oh, I have a better model that's like magically better. And they're like, wrong, wrong, wrong, wrong, wrong. It's just like really like data. And then, but then they were right. Right. So there actually was a sort of a better modeling approach that worked. And I kind of realized, you know, when I was in my early twenties, I was really judgmental of, you know, the people in their late thirties that hadn't like adapted to machine learning at the time, because like rule-based systems were kind of all the rage when a different generation was growing up.

14:11And I was like, wow, you know, I am actually getting out of date myself. Like I'm saying these kind of wrong things that were true 10 years ago and are not true now. And I honestly felt like really bad about myself. And so I did a couple of projects to try to, you know, get up to speed. I started teaching free machine learning classes and deep learning classes to kind of force myself to learn the material. And I actually like interned briefly at OpenAI where I was just like, look, I will just do whatever work you want. I know that I need an accountability partner, essentially, to force me to learn stuff, even though I love to learn stuff.

14:46It's my favorite thing, but I always need accountability partners for anything I do. So I used the students as an accountability partner and OpenAI. And then what was happening was I was showing my old co-founder, Chris, all the cool stuff. And he's a really good engineer. And I'm actually a really bad engineer. I'm really lazy. and like try to write the, like, you know, I'm just like, like people, my co-founders make fun of me all the time for like, you don't really know how Git works. And I just openly, I have no idea how Git works. I just sort of mash the Git keyboard until like, I kind of like, you know, get in a bad state.

15:16And then I like call Chris and beg him to like - The CEO rebased! The CEO rebased! Yeah, I just, I don't know. I don't know. I mean, I don't understand it. And it's like, my co-founders just find it like baffling that I wouldn't understand it. But I think it's like, for them, And, you know, it's like, they're like, wow, this guy like needs some basic tools, you know, like, because, you know, they're like, okay, like reproducibility, like, why don't you just use Docker? I think that's sort of the ops mindset. But I'm like, man, I don't understand Docker. I feel like I installed on my like laptop and then it's always like taking up memory and stuff.

15:48I like, I don't really know what it's doing. And I'm like, kind of scared of it. And like, I don't know. So it's like, I just feel like it's adding weird complexity to understand. And so I think the tools kind of exist in a way, but they just weren't made in a way that like ML people could really use them. Because like, you know, if you're like me, you kind of come from a mathy background or like a research background. You kind of didn't really learn to do like industrial style coding. And so, you know, I think companies have this idea that like the researchers are just going to like throw the thing over the fence and then it's going to be in production.

16:19But it doesn't really work, actually. I think that's a bad pattern that people sort of imagine they're going to do, and they don't ever really do that. You end up with researchers, always the research code bleeds into production in every company. And so I think a better way is to give researchers and ML people tools to just make their stuff more reliable. And it has to be simpler, maybe, or it's just a slightly different audience. You can't just give someone like Docker. You can't just like, you can't, I mean, a lot of people are like, hey, why don't you use like the Git large file system stuff to version your data?

16:55And like, there actually are some reasons, like it doesn't work well with like object stores. So there's some like ergonomics reasons, but it's also just like, man, Git is like complicated. I'm like willing to use it for code. But if you start making me like version my data with Git, like I just want to like cry, you know what I mean? So like, give me something like simple, you know what I mean? where I don't have to like think about it or I'm just going to start like renaming my data sets like latest, latest-really, latest-really for sure, June 27th. So I just need my stuff to be simple. That's kind of the mindset, you know, behind the company is like, let's like make these like kind of simple, clear things that actually help people.

17:31We were talking about how much you wanted to, like you were thinking through how much LLMs were going to change like experimentation and ML tooling when we last saw each other in person, not at the zoo, but before that. And you guys launched this prompt suite in April. Can you talk us through the sort of thought process of, hey, I really admire this. As a leader and as a technical person, you're trying to stay really plastic about what is actually changing in machine learning. How do you think through this change? Well, it's really hard, right? I mean, so what happened was we have a great business that makes a set of ML tools for training models.

18:09And we actually helped most of the LLMs out there were built using weights and biases. And then we started to see, like, wait a second, some of these ML tasks, you could just ask the LLM, right? So instead of doing like a sentiment analysis model, you could just be like, hey, like, is this document positive or negative sentiment? Like for structuring documents, you could just be like, hey, find all the names like in this document. And it actually works super well. And a little piece of me is a little bit sad about that because we have this great, simple, relaxing business that grows revenue every month that I always dreamed of.

18:43So part of me is like, shit, this is actually our first real existential threat, I think. And I went to my leadership team and I went to my board and I was like, I think there's a real existential threat here. And I think they were like, hey, we don't see it in the data. Are you sure? Maybe you're being paranoid. and I guess I do feel sure. And I don't want to say I'm like the only one or like pay myself as the hero. Like, you know, my co-founder is also seeing this and you have people talking about it, but it's sort of like, you know, this threat is like now, right? And we have to actually like get the whole company to do this thing because it doesn't show up in any of our like metrics yet.

19:20But I just really believe that, you know, our customers are rational and they're going to do a thing that like makes sense for them. And so I see a lot of my colleagues being like, oh, there's going to be like lots of different models it's like nice if it were true but like what i see everyone doing right now on july 27th is using gpt like i see like 95 of the people out there you know using gpt for these ml tests and so it's like look we gotta support that and so we really rallied the whole company behind it and uh we pushed out prompts we'd also this is really my my co-founders my co-founder sean had really put a lot of effort into making our stuff really flexible because he's like, you know what, Lucas, there's going to be changes coming.

20:06We don't know exactly what they are, but from the beginning, we really tried to build very flexible infrastructure. So this was a moment where we could really flex that and get out a product for monitoring stuff. And now it's our number one priority is getting out more tools for this new workflow. Out of curiosity, because there's a lot of debate right now in terms of proprietary models versus open source models. And I think there's a really great quote. I think it's from Harrison from Langchain, which is, you know, no GPU until product market fit, right? You should first like figure out if the thing works at all, or if there's a customer need, and that means using GPT.

20:45And then once you prove it out, you know, you may use GPT-4 or something for very advanced use cases. And then you kind of fall back to 3.5, or you start training your own model for things where you just want cheap, sort of high throughput things happening. And it increasingly feels to me like people, the most sophisticated people who are at the farthest sort of cutting edge on this stuff are kind of doing both, right? They use GPT to prototype. And then in some cases, they're training their own instance of Llama 2 or whatever they're using. Do you think that's where the world is heading? Or do you really think things kind of collapse onto some of these proprietary models like over time?

21:17Like it's six months from now, it's a year from now, it's two years from now. I'm just sort of curious about how you think about adoption of open source. You know, it's funny. I feel like lately what I've been telling people is like, I'm just trying to see the world clearly as it is today. I can't predict the future and I can barely keep track of, you know, what people are doing today when I consider it like my full-time job. So I'm like scared to prognosticate like what, you know, might be coming. But I think you're right that that's what's happening now. I think like there are like a bunch of things that could change, right?

21:48Like I think like, you know, GPT is way far out ahead and it's hard to fine tune it. not even possible with GPT-4. And I think that that is like a little, that's not like a technical limitation. I guess sort of like a business model in a limitation. So that might change. I think that there's a lot of hidden costs to running your own model. I think people are really enamored with the idea of running their own model. And I've kind of seen this before where I think at the end people do rational things, but it kind of takes them a while. So I'd rather sort of support what looks like the rational workflow.

22:24I mean, I think the insane thing, must be crazier to be an investor in this world, is like very, very few people have LLMs in production. Like there's probably more companies that have raised money as like LLM tools than companies that have LLMs in production, which is like insane. It's just like an insanely saturated tools market with very few people getting things out. But it's because - Lucas, when you say LLMs in production, you mean my own that I have fine-tuned, that I serve myself. No, sorry. I mean like GPT, like using GPT in production. Oh, really? Okay. Look, I mean, you may be closer to this than me.