Human Data is Key to AI: Alex Wang from Scale AI

0:00There will be a lot more divergence between a lot of the labs in terms of what research directions they choose to explore and which ones ultimately you have breakthroughs at various times. One of the hallmarks of this next phase is actually going to be data production. Basically, no agent really works. Well, in terms of there's just no agent data on the internet, the pricing for model imprints fall dramatically dramatically dramatically automatically. Order is magnitude. Yeah. Order is magnitude. over two years. If you've been listening to the A16Z podcast for a while, you'll know we talk a lot about AI.

0:37We've covered the algorithms of power -alems and the compute required to run them. But equally important is data. Our guest today is as deep as you can get in this world of data, the fuel behind L -alems. In fact, he even recently said, quote, as an industry, we can either choose data abundance or data scarcity. So what did it exist today and what needs to be created, either measured or synthesized? Listen in to find out, as I pass it over to A16Z Growth General Partner, Sarah Wang, to properly introduce this episode. Hey guys, I'm Sarah Wang, General Partner on the A16Z Growth Team. Welcome back to our AI Revolution series, where we talk to industry leaders about how they're harnessing the power of generative AI.

1:22Our guest this episode is Alexander Wang, the founder and CEO of ScaleAI, a company that has become synonymous with Genai and the data needed to power advances in large language models and beyond. With Scale's work across enterprise, automotive and the public sector, Alex is also building the critical infrastructure that will allow any organization to use their proprietary data to build the spoke Genai applications. For those of you who don't know Alex, he is one of the most impressive CEOs we've ever met. And that's saying something, given A16Z first met Alex when he was 21 and already the CEO of one of the fastest growing companies at its scale, which he founded right before dropping out of MIT in 2016.

2:04In this conversation with A16Z General Partner David George, Alex discusses the three pillars of AI, models, compute, and data, and how creating abundant data is core to the evolution of Genai. Alex also shares his learnings from the growth of scale, his approach to leadership, and what he thinks growth -stage founder CEOs tend to get wrong about hiring. Let's get started. As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.

2:45Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Disclosures.

3:02We're very excited today to have Alex Wang, the founder and CEO of Scale AI with this Alexandria Banner. Thanks for having me. I always love talking to you and I always learn a ton, but maybe to start, why don't you just tell us a little bit about what you're building at Scale AI in the multi -win? Yeah, so at Scale we're building the Data Foundry for AI, so taking a step back AI boils down to three pillars. All the progress we've seen has come from compute data and algorithms, and the progress among all three of these pillars, compute has been powered by folks like Nvidia. The algorithmic advancements have been led by the large labs, like OpenAI and others, and data is fueled by scale.

3:38And so our goal is to produce the frontier data necessary to fuel frontier level advancements in partnership with the large labs, as well as enable every to the Enterprise and Government to make use of their own proprietary data to fuel their Frontier AI development. So on this topic of Frontier Data, practically, but how do you actually get it? Yeah, I think this will be one of the great human projects of our time, if that makes sense. And I think that the only model that we have in the world for the level of intelligence that we seek to create is humanity. And so the production of Frontier Data looks a lot like a sort of marriage between human experts and humanity with technical and algorithmic techniques around the models to produce huge amounts of this kind of data.

4:24And by the way, all the data that we've produced today, the internet has looked like that, too. The internet in many ways is this collaboration between machines and humans to produce of course large amounts of content and data, it'll look like the internet on steroids. What happens if the internet, basically instead of just being a human entertainment device with this like byproduct of data generation, what if it worked just this large scale data generation experiment? So you have a very unique perspective into the state of the industry. So how would you characterize the state of models, the language models right now, and I'd love to sort of get into things like market structure, but just sort of what's the state of the industry right now?

5:04Yeah, I think we're sort of closing in at the end of maybe phase two of language model development. I think phase one was the early years of almost like pure research. So phase one hallmarks are the original transformer paper, the original small scale experiments on GPDs, all the way leading up, probably until GPT -3 was this phase one, all research, very, very focused on small scale tinkering and algorithmic advancements. And then phase two, which is maybe GPT -3 till now, is really the initial scaling phase. So we had GP3 that worked pretty well and then opening eyes to start with really scaled up these models to GP4 and beyond and then many companies Google and Thropic meta XAI now many many companies have also joined on this sort of race to scale up these models to incredible capabilities So I think for the past let's say three years.

6:01It's almost been more about execution than anything It's a lot of just engineering like how do you actually have large -scale training work well, how do you make sure there aren't weird bugs in your code? How do you set up the larger clusters? A lot of executional work to get to where we are now, where we have kind of a number of very advanced models. And then I think we're entering a phase where the research is going to start mapping it a lot more. Like I think there will be a lot more divergence between a lot of the labs in terms of what research directions they choose to explore, and which one ultimately have breakthroughs at various times.

6:31And it's sort of an exciting alternating phase between maybe just raw execution versus sort of a more innovation -powered cycle. They've kind of gotten to a point where I wouldn't say there's like abundant compute, but they've had enough compute that they've needed in order to get to the models where they're at. That's not a constraint necessarily. They've kind of exhausted as much data as they possibly can on the frontier labs. And so the next thing will be breakthroughs on that and then advancing the ball on the data side, is that fair? Yeah, I think basically, yeah, if you look at the pillars, compute, we're all continuing to scale up the training clusters.

7:07So I think that direction is pretty clear on the algorithms. I think there's to be a lot of innovation there. Frankly, I think that's where a lot of the labs are really working hard. I think on the pure research of that. And then data, you can have alluded to it. We've kind of run out of all the easily accessible and easily available data out there. And yeah, coming across all done, everybody's got the same access to it. Yeah, exactly. And so a lot of people talking about this as the data wall, we're kind of hitting this wall, where we've leveraged all the publicly available data. And so one of the hallmarks of this next phase is actually going to be data production.

7:38And what is the method that each of these labs is going to use to actually generate the data necessary to get you to the next levels of intelligence, and how do we get towards data abundance? And I think this is going to require a number of fields of advanced work and advanced study. I think the first is really pushing on the complexity of the data. So moving towards frontier data. So a lot of the capabilities that we want to build into the models, the biggest blockers actually a lack of data. So for example, agents has been the buzzword for the past two years, and basically no agent really works.

8:12Well, in terms of there's just no agent data on the internet, there's no just pool of really valuable agent data that's just sitting around anywhere. And so we have to figure out how to produce a really high quality agent data. Give an example of like what would you have to produce? So we have some work coming out on this soon, which demonstrates that right now if you look at all the frontier models they suck at Composing tools. So if they have to use One tool and then another tool. Let's say they have to look something up and then write a little Python script and then Chart something they use multiple tools in a row.

8:42They just suck at that They just sure really really bad at utilizing multiple tools in a row and that something is actually very natural for humans to do Yeah, but it's not captured anywhere, right? That's the point right exactly. You can't actually go Take the capture of somebody going from one window to another into a different application and then feed that to the model so it learns Right, so yeah Yeah, so so these sort of reasoning chains through when humans are solving complex problems We naturally will use a bunch of tools will think about things will reason through what needs to happen next We'll hit errors and failures and then we'll go back and sort of like reconsider A lot of these reason chains, these agentic chains are the data just doesn't exist today.

9:21So that's an example, something that needs to be produced. But taking a big step back, what do you say happened on data? First is increasing data complexity, so moving towards frontier data. The second is just data, but it's increasing the data production, capturing more of what humans actually do in the field of work. Yeah, both capturing more of what humans do, and I think investing into things like synthetic data, hybrid data, so utilizing synthetic data, but having humans be a part of that loop so that you can generate much more high quality data. We need basically, just in the same way, I think with chips, we talk a lot about chip foundries, and how do we ensure that we have enough means of production chips.

9:54And the same thing is true for data. We need to have, effectively, data foundries and the ability to generate huge amounts of data to fuel the training of these models. And then, I think the last leg of the stool, which is often rated as measurement of the model. And ensuring that we actually have, you know, I think for a while, the industry is just sort of like, We just add a bunch more data and we say, how good model is and we add a bunch more data. We say, how good model is. But we're going to have to get pretty scientific around exactly what is the model not capable of today. And therefore, what are the exact kinds of data that need to be added to improve the model's performance?

10:26How much of an advantage do the big tech companies have with their corpus of data versus the independent labs? Yeah, well, there's a lot of regulatory issues that they have with utilizing their existing data corpuses. You can look through, this is before all this generated V .I. work, but at one point, meta did some research that utilized basically all the public Instagram photos along with their hashtags to train really good image recognition algorithms. They had a lot of regulatory problems with that in Europe. Like it turned out to be a huge pain in the ass. So I think that that's one thing that's kind of difficult to reason through, which is to what degree from a regulatory perspective, particularly in Europe, these companies are going to be able to utilize their data advantages.

11:07So I think that was kind of TBD. I think that the real way in which a lot of the large labs have just Dramatic advantages is just they have very profitable businesses that I can provide Near infinite sources of capital for these AI efforts and I think that that's something that I'm watching pretty intently Or I'm very curious to see how it plays out. There's this whole question the industry is like are they over -investing? And if you listen to their earnings calls at the big tech companies, they're like look our risk is under -investing not over -investing What do you make of that? Yeah, I mean, if you think about let's take the incentives of any one of the CEOs of the, put yourself in the shoes of Sundar Pachai or Mark Zuckerberg or whatnot.

11:48And Sontia or Sontia. And to your point, if they really nail this AI thing, they could generate another trillion dollars of market cap, probably very easily. If they really are ahead of the competition and they productize in a good way, the trillion dollars of market cap cut them no brainer. And if they don't invest the extra whatever it is 20 or 30 billion of CapEx per year and they miss out on that and then there's some real existential risk I think too for each of the large yeah for each for you in each for me. Yeah, all their businesses are potentially deeply disruptable by AI technology. So the risk reward for them is very obvious.

12:22So that's I think the big picture thing and then from a more tactical level I think all of them are going to be able to pretty easily recruit their capitalist investments just by by worst case making their core businesses more efficient and effective. So for example, like, you know, you got GPU utilization for Facebook advertising. Yeah, Facebook, Google, they make their advertising systems a little bit better. They can recoup billions of dollars just by better performance there. Yeah, Apple can easily recoup the investments if it drives an upgrade cycle. I mean, these are things that I think are pretty clear.

12:54Look, it's generally great for the industry that they're investing so much capital because they also are in the business of renting this compute out, or at least in the case of Google and Microsoft they are. And the models are making their way like Lama 3 .1 is open source. And so even the literal fruits of all the investment are becoming broadly accessible. And so the surplus generated from the open source and these models is kind of insane. It's insane. Okay, so that's a great segue into market structure at the model layer. So what do you think actually happens? Are there the few players that we've all identified now, the handful, and they all compete?

13:27Do you think it's a profitable business? What impact does open source have on the quality of the businesses? Take us a couple years ahead and give us your forecast. Yes, we've seen over the past even just like year and a half the pricing for model inference fall dramatically dramatically dramatically like order negative. Yeah, two orders of magnitude. Yeah, two orders of minutes over two years And so it's this shocking thing that it turns out intelligence might be a commodity But no, I mean, I think that this huge sort of lack of pricing power, let's say, on the pure model layer certainly indicates that renting models out on their own may or may not be the best long -term business.

14:08I think it's likely to be a relatively mediocre long -term business. Well, I guess it depends on the breakthrough thing, which is the earlier point, right? To the extent that someone actually has a durable breakthrough or multiple people have durable breakthroughs, like them potentially market structure is different. So two things. If Medicare continues open sourcing, that puts a pretty strong cap as to the value that you can get from the model layer. And then two, if at least a handful of the labs are able to have similar performance over time, then that also dramatically changed the pricing equation.

14:39So we think that it's not 100%, but chances are the pure model renting business is not the highest quality business where there are much higher quality businesses are going to be above and below. So below, I mean, Nvidia is obviously an incredible business, but the cloud's also have really great businesses too, because it turns out it's pretty hard Logistically to actually set up large clusters of GPUs And so the cloud providers actually have pretty good margins when they rent out and the traditional data center business is very much a scale game Yep, right. So they are massively benefited relative to smaller players.

15:11Yeah, exactly So I think picks and shovels. So if you're under the model layer, I think there's great businesses there And if you're above the model, if you're building applications, ChatGapT is a great business. And a lot of the apps in the startup realm actually are working pretty well. I mean, none of them are quite as big as ChatGapT, obviously. But a lot of apps, if they nail the early product market fit, end up being pretty good businesses, great businesses as well. Because the value that they generate for customers, if they get the whole user experience correct, far exceeds the inference cost of the models.

15:42There's some cool stuff here, right? I think on Anthropics launch of artifacts in cloud. It's like the first pin drop of this major theme of all the labs are going to be pushing much deeper product integrations to be able to drive higher quality businesses. So that will be the other story is I think we're going to see a lot of iteration at the product layer and the product level. The sort of boring chatbots is not going to be the end product. That's not the end all of you. It's disappointing outcome. Yeah, exactly. And product iteration and the product innovation cycle is very hard to predict because I mean open I was surprised how popular touchy pt was I don't think it's like super obvious to me or anyone in Industry frankly what exact products are going to be the ones that hit and what's gonna provide the next legs of growth But you have to believe that an open a iron and thropic can build great Applications businesses to for them to be long -term.

16:33Yeah independent sustainable. Yeah for sure Yeah, and then it's what drives competitive advantage. Obviously you have the model, a tightly integrated product on top of it, and then the good old fashion modes from there. Yeah. Work flows, integrations, all that stuff. I think you can clearly see that, I mean, both OpenAI and Enthorobic Heart, she's product officers within, I don't know, two months of each other. Yeah, that's a good idea. And then, like, it's sort of a change of tune, where they're like, I know, we're very purely focused on this, and it's okay. I think there's the realisation to it.

17:03So yeah, exactly. It makes full sense. You've got an application business with some really interesting customers. What are you hearing from enterprises as to how they're actually putting this into place? I think what we've seen is there was a huge amount of excitement from the enterprise. A lot of enterprises were like, shit, we have to start doing something, we have to get ahead of this. We have to start experimenting with AI. I think that that led them to this fast POC cycle where they're like, okay, what are all the low -hanging fruit ideas that we have? Go buy AI stuff. Yeah, yeah, and let's go try all of it and some of those things are good some of them aren't good But I think regardless it's been this big frenzy much fewer of the POCs have made it to production Then I think the industry overall expected and I think a lot of enterprises are looking at now and the doomsday That they thought might have happened hasn't really happened AI has not fully Terraformed and transformed most of the major industries like it's not like totally, you know It's sort of marginal stuff.

17:57It's like efficiency gains in support and then some of the creative tasks and things like that. Yeah, exactly. Otherwise it's pretty light. The thing that we think a lot about is like, what AI improvements or AI transformations or AI efforts that we're working on actually can meaningfully drive the stock price of the companies that we're working on. And so that's what we encourage all of our customers to really be thinking about because at the end of the day, the potentials there. There's lead and potential for almost every enterprise to implement AI at a level that would meaningfully boost their stock price, mostly in the form of cost savings, efficiency gains.

18:31Well, today in the form of cost savings, but then also much better customer experiences. Like, I think at a lot of industries where there's a lot more manual interaction with customers, you should be able to drive much better customer interactions. If you have more standardization and you were able to use more automation, and then those eventually would make their way to gains of market share with respect to competitors. So that's what we're pushing our customers towards. And I see it, some of the CEOs that we work with, they're all on board, and they understand that it's gonna be a multi -year investment cycle.

18:59They might not see gains the next quarter, but if they actually pull through the other side, they're gonna see massive transformations. Yeah. I think that a lot of the friends around small use cases and sort of the more marginal use cases, I think that's good. I think it's exciting. I think they should be doing it, but to me, that's not what we're all here to do. Yeah, it's very much like the application layers, There's very much like phase one right now, which is, yeah, there's some automation, but it's largely like chatbots. My hope as a startup investor is that over time, there's a window that opens for the startups where product innovation will help them to win and beat the incumbents.

19:33My partner, Alex Rampel, is this phrase, which is, is the startup going to get to distribution before the incumbent finds innovation. I think there's an opportunity for it, but it's like the tech is too early right now. Yeah. I don't know if you would agree with that. But I think the tech is too early to imagine. Yeah, again, because it's mostly cost saving. I think if most of the benefit is on the cost saving side, then that's not really enough to disrupt large incumbent that has already pushed their way through all the costs of growing distribution. How valuable do you think is the data inside of an appraisal?

20:03Like you've said, though, JP Morgan has whatever, 15 petabytes of data, so I'm like, I never wrote the numbers. But is that overrated? How much of it is actually useful? Because today, most of that data has not given them some meaningful competitive advantage. So do you think that actually changes? I think AI is the first time you could see that potentially change. Because basically, obviously, there's the whole big data wave. Big data boils down to better analytics, which is helpful, like marginally helpful for business decisions you're making, but not deeply transferring. It doesn't massively change the way the products work.

20:34Yeah, exactly. Whereas now you actually can imagine some massive transformation in the way the products work. So let's take it any big bank. A lot of the valuable interactions between a user and a large bank, like a GP Morgan or Morgan Stanley or whatnot are human driven, are people driven. And they try their best to ensure that the quality of experience is very high across the board. But obviously with any large process, there's only so much you can do to assure that. But all of your prior customer interactions and all the ways in which your business has worked historically is the only available data to be able to train models to do well at this particular task.

21:13And if you think about wealth management, there's very little indistribution data that on the internet that you could trade a model off of. So there's behind the walls, there's actually quite a bit, it's very rich. Yeah, huge amounts of data. So I think that a lot of the data is probably not super relevant to actually transform your business, but some of the data is hyper -valuable. So I think enterprises have a lot of trouble and challenge around actually utilizing any amount of data that they have. It's poorly organized, it's sort of all over the place, they pay, consulting firms, tens of millions of dollars, hundreds of millions of dollars to do these data migrations.

21:45Even after that, no change of results. Yeah, no change of results. I think it's historically very difficult place for enterprises to really drive transformation. In some ways, this is the race. Are they going to be able to figure out how to utilize and leverage their data faster than some startup figures out how to somehow get access to the data. possibly different product with a little bit subset of the data. Yeah, exactly. Shifting gears to how you run your company and how you built your company. One of the things that you've talked about is a mistake that you made during the go -times of 2020 and 2021 around hiring.

22:22And this notion that in order to scale, you had to hire a ton. And it's something we saw with all of our portfolio companies. It was like, hey, there's war for talent. And it meant that we got to go higher, we got to go higher, we got to go higher. So what were the lessons that you learned through that process and then how have you changed how you've done things afterward? So over the past few years, we've basically kept our head count flat. I mean, we've grown it very slightly as the business grown, but the business itself is 5x, 2, 6x, you know, the business has grown dramatically and the takeaway from this entire process is it feels very logical that more people equals better results and more people equals more stuff being done.

23:03But rather paradoxically, I think if you have a very high performing team and a very high performing org, it's nearly impossible to grow it dramatically without losing all of that high performance and all of the winning culture. Yeah, reducing the communication and coordination overhead actually increases productivity. That's definitely true. And I think it's actually something even deeper, which is that a very high performing team of a certain size is almost like this very intricate sculpture in this interplay between all the people in the team. And if you just add a bunch of people into that, even if the people are great, it just screws the whole thing up.

23:37And no matter what as you add people, you're going to have regression to the mean. You know, if you kind of observe companies that do scale how cataloged and that's pretty core to their financial results, I think they acknowledge that regression, that mean regression. So if you think about the scaling of large sales teams, for example, yeah, sure. You acknowledge that you're going to have that mean regression, but you just operationalize so that you're like a little bit above the mean. And if you're able to do that, then the whole equation still works financially. I'd say sales is different than the product.