The AI Race Has a Leaderboard | Arena CEO Anastasios Angelopoulos

26:38Do you know something that I don't know? Yeah, I do. I do. You do? I know what's happening in Southern West. They're making aliens. We've had contact with RSI. RSI. RSI is here. No, you know more than me. I know more than... But yeah, I think that's number one. It's just so cool to be part of this. Mm-hmm. And then there's things about being a CEO that I love. I would say I love working with people, a diverse array of people from all sorts of walks of life. You know, as an academic or as a PhD student, you interact with technical people, people who are experts at their craft, really honest people, sweet people that have no idea about how necessarily to productionize something, but that are brilliant at what they do.

27:28And I love those folks, but it's only like a tiny, tiny slice of the world. And now I work with BD people who everything is a negotiation. And I work with, you know, uh, like lawyers, and I work with investors, and I work with, uh, you know, engineers who are like great at building things and taking something from an idea into a product that you actually love to use. And I work with marketing people. And all of the people that we have at Arena are absolutely best in class. They've built generational companies before. Yeah. And so it's really cool to be just, you know, working with so many amazing people.

28:03And then people have problems. And that's my job as CEOs, deal with people's problems. I'm sure you guys know. You know, you're really good at it. Actually, you're exceptional at it. From the basement as a PhD student to a year later, running a company at this scale, I think just from point A to point B, I don't think I've seen a slope higher than yours. That's so sweet. I really appreciate that. You've seen a lot of slopes. So that's a high compliment. But I'll tell you, the people problems, Of course, they take up more and more of the time as time goes on, because everything ultimately is a people problem.

28:39A company, you design the company, the company builds the products, right? So, I mean, there's some truth to that. I love being part of the details, so I'm always gonna be part of the details, but at the end of the day, you need to make sure the right people are in the right place at the right time, and are pointed in the right direction to make things happen, and that they're not caught up with, like, dumb crap. But I kind of love dealing with people's problems, too. And you just gotta love people. You're like, oh, this person's having this problem because they are emotionally feeling this. And oh, how can I help them fix this insecurity that they have?

29:12Or this interpersonal, this argument that's happening because it's something stupid. How can we help resolve that? And I kind of relish that. Now there's parts I don't like. Parts that I don't like about being a CEO, primarily that my calendar is just filled with meetings. I just have like 10 hours of meetings a day. I do 10 hours of meetings i go home i do whatever work i couldn't do in the meetings and then i rinse and repeat meetings are not always pleasant sometimes you just want to be a little like i love doing technical work too um so i try to carve out five to ten percent of my time to do that but you know that's always last last on my priority can you steel man why you think this is more important than electricity?

29:58Like, I'd be very curious your argument for why. Okay. So why is it the most transformational technology that humans will ever invent? Obviously, like electricity - Ever invent or ever have invented? Ever have invented. Yeah. Ever have invented. Obviously, electricity is foundational, right? We can't have AI without electricity. So in some sense, like if you think about it hierarchically that way, electricity would be more important. Now, the counter argument to this would be just in terms of the realized impact, that sort of the incremental value that AI would add would be greater than that of electricity.

30:34And so why would that be? AI has the potential of literally creating intelligence on demand, which is something that humans have never seen before. and could be influential insofar as it actually even deprecates humans. They could literally make humans obsolete in the sense that we're able to create an organism that can pursue any goal faster than we can, that can create productivity at a greater degree than we can, that can explore the universe more efficiently. And the impact of that, I think, sort of if you were to try to decouple everything else electricity can do versus everything AI can do, that impact may be greater than electricity, because it will just supersede everything that humanity has ever accomplished.

31:35That would be the argument. So to be clear about the thesis, it would be, if you take everything that electricity has done and will do, outside of AI and compare it to the impact of AI, then the latter will be a greater. I'm curious, what's your reaction to that argument? It's beautifully said, and I fully agree with you. There's nothing to disagree on. I think, you know, we've talked about this for the last few years, that this is the coming together of all the pieces of technology that have been created over the last 60 years from the transistor in silicon to firmware software operating systems the internet mobile cloud everything pulled together allows this moment to exist yeah right so it is the it is the coming together the culmination of everything's happened in our industry ever since Our firm was started 55 years ago, and it is greater than all those combined.

32:47And I actually love the framing of if you took electricity and took to one side what electricity has done for humanity and put to the other side what AI will do for humanity, and the way you said it, the latter will be greater than the former. Yeah, no debate. no debate no debate do you have a different opinion no i mean i hope you're right because something's got to justify these these valuations so so you know i think the real question is what happens to money yeah will money be so it'll it's interesting because money there's there's something that has made capitalism work so far which is that capital and labor are coupled like you can earn money by through labor and that is no longer really going to happen right so So that's where this capitalism stops working for people.

33:39And then you have to start thinking, what's going to happen to people? I think that's the fundamental question is the decoupling of labor from capital. Yeah. Very fundamental. That is really the crux of the matter for humanity is the decoupling of the two. Historically, the stock market revenue for companies, which is tied to the value of the stock market, was tied to the labor. Yeah. Input was labor. And as labor becomes detached from the creation of value, the stock market capital value diverges from labor input. Totally. Right. And that's really what we have to all grapple with as a society over the next decade or so, which I think we have to flush this stuff out.

34:26um there's just one one stat that really jumps out um is if you look at the overall market cap of um the stock market it's like i think the u.s stock market's like worth um 100 trillion dollars something like that in that order roughly and uh it's it's sort of in line with the the gdp of the world so in like 20 30 years ago uh the overall value the stock market was way lower than the overall gdp of the world so capital value has appreciated a faster rate than gdp yeah and it's going to accelerate in that way um and it's already actually if you look at the u.s stock market um a lot more of the value accrues to um the delta is like the u.s gdp is like 22 trillion the u.s stock market's like 60 trillion so it's like a 3x yeah and the overall worldwide is like a 1x so the u.s is already at a 3x capital value to overall gdp totally and that's going to accelerate that's going to crazy degree right and so it's the separation of labor from capital.

35:42And so I think that is what we all have to sort of grapple with as a society is how do we live in this world where more and more of the value of the world will accrue to potentially fewer companies. Totally. And also how do I think the idea of how we assign value is another thing that we are going to need to think about. because the assignment of value comes from scarcity and scarcity comes from the coupling of capital and labor the fact that you can only make x dollars because you own a salary and therefore you have to spend that money carefully and therefore value is a signal like price is a signal value right the whole point of like this these economic theories is that of capitalism is that price is information.

36:33The price is the information about how valuable something is. And that information is driven by scarcity because the sort of like decentralized market of consumers of whatever that product is, they go and they pay whatever it's worth to them as opposed to other goods. But what happens when those two things are decoupled? And I think there's two ways of seeing it. One way is what happens to humans? How do humans assign value in a world where, let's be honest, there's going to be a big fraction of people that just can't make money? And then there's another side of things, which is agents. What if agents can assign value?

37:11Do they have the right to assign value? Is value just something within the construct of a human? If you're making life better for a human, that's valuable. Or is there a construct under which you're making life better for an agent? and therefore an agent with a wallet can pay for that, we call that value, and that the money that the agent is spending doesn't need to come from a human. So can agents autonomously be earning value? And that's something we're going to need to decide as a society is whether or not we're even going to allow the possibility for autonomous agents to directly assign value themselves.

37:51Is all of this assuming that the models will continue to get better at the rate that they have? Yeah, models are harnesses. Not necessarily even at the rate that they have. This might happen in 20 years. It might happen in two. My guess is it's going to happen closer to two than 20. So I think we better start thinking about it now. Why is that your guess? It's just the rate of improvement hasn't decreased. If you look at the arena, we're seeing, at times, bigger gaps between successive generations of models than we have in history. still to this day. You're saying that the newer models from Anthropic and OpenAI are taking bigger step function leaps and improvements in your leaderboard than like GPT 3.5 to 4.

38:40Yeah, it's happened before. I mean, GPT 3.5 to 4 was another one of the biggest leaps in history. But we are still seeing some of the biggest leaps in history to this day. So for example, GPT Image 2 versus 1.5 was, I think, the biggest gap that we have ever seen in history on ARENA. It was nearly a 100 % win rate. And by the way, it's almost impossible to get people to agree on anything. Like, it's almost impossible to get a large number of humans to vote the same way on any political issue, on, you know, basically anything at all. But 100 % of them are voting GPT Image 2 better than every other model that we had ever tested at the time that it was released and we're seeing phenomena like that still happening so yeah i don't think and rsi you know it's the buzzword of the day if this happens um we'll absolutely be seeing acceleration question for you what are you long and what are you short i think that the api layer people are assuming a degree of lock in the apis that i don't think is necessarily going to exist.

39:46Anthropic revenue has been booming, booming, booming on the API. But to some extent, it's also easy come, easy go. It's like if you are spending that much money that quickly, there's no chance that you're not able to switch vendors. And so if an open source model comes by and it is able to do 90 % of the work at 10 % of the cost, which is totally feasible that that might happen over the next couple of years, and that certain businesses might have all their capabilities be able to be done by the mini light version of whatever the next Kimmy is? Well, I think that's a pretty structural problem. And so what are people doing?

40:21What are labs doing to do this? You know, OpenAI Anthropic, OpenAI has this huge consumer business, which is kind of like going to be able to keep it afloat anyway. By the way, one thing I'm long is the consumer ad marketplace on OpenAI, which has obviously not been fully utilized to this day, but I think it remains one of the biggest opportunities in AI. Because the question is, how are you going to generate hundreds of billions of dollars in revenue for them? Right? like annualized. And there's like only a few businesses that know how to do this. And it's like ads and great hardware businesses.

40:54And I actually don't know anything else. You probably know better than I. No, those are your, you know, take Google. You take Amazon. You take NVIDIA. Exactly. Right? Tesla. Tesla. You know, consumer or great hardware. And like ads or hardware. If he's right, let's just imagine like if open source... delivers, breaks out. Do you think that value goes up the stack? Like, let's just say right now, so many of our application layer companies have very expensive costs to serve. And if you got rid of the gross margin problem, these would be some of the most incredible businesses of all time, like ever.

41:37So I guess if open source can alleviate the gross margin problem because their cost to serve goes down by 80 to 90%, then naturally, does the logic follow that value continues to go up the stack? I think that's been our fundamental belief here, that we're at maxis here because we believe that the models will continue to get better, the open source models will be even better, and the cost will come down. And we already see that the army of lawyers that AI lawyers, or whatever we're going to call it, at Harvey is the use just continues to go through the roof or take any app layer company. There's an insatiable demand for the apps that get the job done.

42:24And you don't want like the half job done. You want like all the particular needs of the legal industry met or the medical industry or the coding. So, yeah, fundamentally, like, all the things we're talking about benefit the appellator companies. And that's always been sort of the, that's how it's always played out in every sort of cycle of the industry, is that companies, big companies especially, buy from other companies that package up a really nice solution, you know, show up with the, the person who's going to help them implement it, get it up and running, getting it, getting the value delivered to the end user.

43:11And that usually comes in the form of an app layer company doing that for them. And so I think the tailwinds for these app layer companies are totally there. I know they were called rappers for a very long period of time. And I think that's, it's always the, you know, the very lazy way of, you know, identifying these companies or uh describing these companies uh but but yeah i'm i'm in the the camp of there's just a lot of value that will accrue to these app layer companies well i think that this is a really interesting point so then how do you view the lapse it's to your point like you know it especially as open source unless like we all just lock down open source from china right Which, I don't know.

43:59Regulatory capture. Yeah, do you think that happens? I think that there will be serious regulation on open models, unfortunately. Yeah. Yeah, and I don't know exactly what shape that'll take. I think in part, it's being shaped right now, and that people like you guys probably have the power to make sure that's done responsibly. Right. But I would predict that we regulate those. There will be regulatory capture. Yeah. Maybe the silver lining is then we get a good open source model here. Yeah, I think that that's the saving grace. And to be clear, I think the commercial incentives favor this anyway, because we are going to need American models so that people can build on a reliable supply chain and also one that doesn't have a tax factor, frankly, a foreign adversary.

44:43So we're going to absolutely need American open source. Maybe this will accelerate it. But I just hope the regulation doesn't slow things down, too, because it could also slow things down. Would that be the worst thing for your business? Oh, for sure. Our business sucks if we only have one model provider, you know? Right. And if we have two, we're barely making it.

45:06We want 10. We want 50. Um, yeah, I think we could probably do with three or four. Um, but you know, regardless of our incentives, it's bad for everyone. I mean, who, who wants an oligopoly? Like, the whole point of this, like, capitalist system is that we should try to encourage competition. And so now, if we have the winners going to Washington and convincing politicians that we should create these regulations that are going to be harder for, you know, that basically only incumbents are going to be able to satisfy and make it harder for other upstarts to be able to increase adoption and get their models out there, I mean, come on.

45:46Like, this is the most anti-competitive possible behavior. yeah you know um i was listening to um brett taylor we're at some thing and he was talking and like i would consider brett to be more in the know than like anybody and like about as competent as competent like whatever competent is that's him like he is i have so much admiration for him and obviously he's on the board of like he knows more than most of us and whenever he talks about anything AI related, he's like, listen, I have no idea. Like when Brett talks, he's like, I genuinely have no clue. Cause he's like, I've never seen anything like this.

46:28And he's like, I know more than anybody in this room and I have no idea. And I'm like, you know, I kind of respect that, you know, I'm just like, it's just, we've never seen anything like this. Yeah. We've never seen it. It's insane. Yeah. Completely. Going back to the, like running your company for a second. Sure. How do you think about, like if you did 100 million in eight months, I don't know what the goal is in a year from now, but - We don't either because we keep beating it. A lot more. We keep setting it, we keep beating it. How do you support the company? Like how do you, I don't, how do you support revenue growth that is going parabolic like that?

47:11Like from a company perspective, Like, how do you even build the scaffolding around a company that's growing that quickly? Yeah, I mean, the good thing is that I think AI has made it easier to run a very high revenue company with fewer people resources. So we don't need to scale people linearly with revenue. Fortunately, we've avoided doing that entirely. I would think about it more like, how do we put the right people in the right place at the right time and make great products? and how do we attract the best people in the world towards a small team that can support huge amounts of revenue. And of course we need to fill in to certain G &A things as we go.

47:53You know, we probably need more lawyers because we have more contracts. The number of contracts and number of lawyers kind of scale together. But it's not true of the raw revenue. Yeah, or the product. You know, the product runs on its own. It's arena.ai. It's this website. It has all these users. You know, of course we want to keep making the product better for them. We want to keep refining the user base and, you know, making the users better, giving them more tools, making it safer, decreasing abuse of the platform and so on. So we want to hire for all these things. We have our enterprise play.

48:24We want to hire great infrastructure engineers for. But it's not like that's a revenue scaling problem. It's more of a product scaling problem. What's your worst day at the company so far? Man, worst day at the company. Such a positive guy. I know. Well, it's easy to be positive when you recruit 100 million in eight months. No, there's some low periods. Yeah. You know, at the beginning, in the early days, there, um, it's so hard for me to talk about. And there was this moment. Uh, we were just, you know, in the early days of trying to start the company, trying to get it off of the ground. And we were still supporting labs and open source and other academics.

49:11sort of on an ad hoc basis, giving people data, helping people do their research. And we were about to raise our seed, which was, of course, stressful for me. I had no idea how to do it. Yawn was helpful. You, of course, supported us in that time. And a very stressful period for me. And then my father-in-law passed away suddenly. He passed away of just a heart attack, I guess, in Serbia, my wife's servant. And so I kind of had to drop everything and go to Serbia. He was also her person. They were so, so close. And they're similar people. Like, when I talk to her, I see him. And I was close to him, too.

49:57I really loved him. And so he died literally while we were in the, hey, term sheet's coming. How are they? Stage of our seed. And so I had to call people like Moon and say, hey, guys, I need a few days. And then we went through the funeral. And then two days after the funeral, I was on calls again. Like negotiating the deal. Right? Because I needed to, because the company needed me to do that. And maybe I didn't 100 % need to at the time, but I thought I did at the time. That people wouldn't wait. And that the deal was hot. And I didn't know what to do, but I knew that I needed to do what's right for the business, because there's people's livelihood relying on me.

50:37And then after that, you know, I think it took a long time for my wife to forgive me for the fact that I had to go be there during that period. And, you know, she was mourning so deeply. And I would say that she cried every day for many months. And then eventually, it was every two days. Maybe six months later, every two days. And then three months after that, it was every four days. And now she cries every so often, she says, I remember my dad. And I'm starting to forget my dad. I forget the sound of his voice. And that's just so difficult. Mamoun knows, because he's been through a similar experience.

51:25Yeah. And then, on top of that, a couple weeks later, there was this awful moment where there was this paper that was released called Leaderboard Illusion. People don't know this about Leaderboard Illusion. You guys might have seen it. Not only is it filled with a bunch of lies with data, basically incorrect analysis about ARENA, that sort of is, you know, it's this coalition, MIT, Stanford, you know, whatever. So it looks like it's a reputable source. But actually, when you dig into the analyses that I had about Arena at the time, it was about, oh, this company is favoring closed-source models over open-source models and blah, blah, blah from Cohere.

52:08Of course, Cohere was like number 70 at the time. So, you know, you say what you want about the incentives. We've always been fair to everybody. And that was very difficult because it felt like a stab in the back from some people that could have been my future colleagues if I ever decided to go to Stanford. Yeah. And... was another difficult experience on top of the sort of personal challenges that we had at the time. It was very hard for me and my wife, and it happened a few months before our wedding. And so we thought that our wedding was gonna be a sad time. We thought that we were not gonna have a good day.

52:44But, you know, we ended up pulling, she ended up kind of like rallying for that day, and we both had an awesome time at our wedding, thank God, because it's only something that you get once. but there have been some very low points in arena. Startup-like, the lows are low and the highs are high. Hi, Goosebumps. Thank you for sharing that. Of course, yeah. Can I ask you about, can I ask another question? Of course, yeah, yeah, yeah, yeah. I'm an open book. When you, I thought the framing of how you said it took a while for her to forgive you for that. Was it her forgiving you or was it you forgiving yourself?

53:20It was probably both, but she definitely needed to forgive me. Yeah, I think it's for her - Prioritizing the company? Yeah. Right. Right. I think it felt like a betrayal to her that I would prioritize in the way that I prioritized. And probably she also had more clarity of thought and was a little further away from the fundraising details for me to say, hey, maybe I get why you would do this meeting, but I don't get why you would do this other meeting. and like not all the decisions i think made sense to her probably because she was right and they didn't all make sense to be honest i don't know for what it's worth um i got to see a very small maybe like 0.01 of you during that time in terms of what came through to me but the level of care and love that you showed towards her and your father-in-law was actually unique to hear from a founder CEO who's building something in AI there's a level of humanity and empathy and compassion that you had towards others that you actually don't see from founder there's so much egocentricity.

54:39So it made me, and we'd just gotten to know each other, respect you so much as a human being, but like someone who is more than just a PhD, Stanford professor, founder in AI. It was just like, and maybe that's why from point A to point B, you have this slope because you may be the full package. i appreciate it no i think we really resonated together on that personal on that personal note that we've been through some similar difficult experiences and um yeah for for better for worse i have a big heart um so i you know i appreciate that you value that man thank you for sharing that of course um what does the rest of this year look like for you and the company yeah a lot of big things coming for Arena.

55:34Two big priorities, I would say. The first is agents, and the second is building out the enterprise side of our business. So on agents, you know, we've been historically known as a platform for evaluating models, but when you get into agents, it's just such a more heterogeneous ecosystem. It's not just the model, it's also the harness. And the harness can be very, very difficult to, you know, evaluate. It can be a multi-component system. It includes a a bunch of sub-models and sub-agents. And it's something people don't know. You can't go to somebody today and say, hey, which model, which harness combination is the best?

56:14But we can actually do that at Arena. And so what we do, if you go to arena.ai slash agent, we have agent mode, where you can do coworking-type tasks, like cloud coworking-type tasks on Arena in browser. You can connect your code base, you can do everything. And when you type something that will basically give the agent a computer and it will go in and it will allow you to do, you know, open-world tasks. And as part of that, we can randomize both the orchestrator and the harness for the agent, so we can collect all of those interaction effects that make it challenging. The goal being that people shouldn't really have to think about what harness to use or what orchestrator to use.

56:50It should just be an automated system. We can even do that via routing, which we've also built. We've been building routers for the last year. Routers are now hot. But we were in the routing game. So hot. I know. So hot. We've been in the routing game for quite a while. And then the second would be, how can we take these ideas, the sort of like self-improving data flywheel that we built on Arena that allows us to build the best routers in the world and then copy paste them into every business, help businesses take better advantage of their organically occurring data so that they can build a better routers, avoid vendor lock and get AI sovereignty and so on.

57:21And so those are the two big priorities. Are you hiring towards both of those priorities if people are listening? What are you hiring for? Infrastructure engineers, AI engineers, those are probably the most valuable. If you can come help us build out our agent product and our enterprise product, which requires really high availability, high uptime, you know, this routing functionality, data engineering, that's all really important. Machine learning research. You know, I come from a research background myself, so I know a lot of ML researchers, but if there's people out there that are interested in these fundamental problems around, how do we optimize, basically?

57:58How do we measure? And then how do we optimize? Because you need to know what target... Basically, you need to know the target in order to optimize it. And that's what evaluations are all about. I view Arena as being the system of value. That's where I see us in the ecosystem, the AI ecosystem. We can help developers, businesses, and the whole community define value through the real world. and then once you have a value system, then you can hill climb it. And the whole game is about picking the right value system. And so our value system is a utility to real people. And so anybody who's passionate about that mission should come join us.

58:44Thank you for doing this. Of course, Will. Anything else for you? Last one for me then. When you hear the word grit, what do you think of? Oh, grit. I think of people that never give up. I think of people that day in and day out, they are coming in and they have their nose to the grindstone. And there might be terrible things happening around them, or they might be pulled in other directions, but they're never going to lose focus. They're always going to pursue the goal, and it's never going to be about themselves. They're never going to have any ego about it. Grit means somebody who um is always putting the greater mission ahead of themselves and is willing to do anything to get the job done my um i was going through hell and back on the company over the last couple of weeks on some stuff and my fiance on the mirror of our bathroom wrote the man in the arena quote yeah the full teddy roosevelt man in the arena quote just literally in marker on the entire on the entire mirror.

59:53And yeah, it reminds me of what you said. Like, sometimes you just got to be in it. Totally. Appreciate you doing this. Thank you so much. Amazing. Yeah, thanks for being here, guys. That's it for now. If you liked the episode, please leave us a review or go back into the archives where we've done more than 200 episodes with some fantastic folks. This podcast is a Kleiner Perkins production and I'm Juven. Thanks for listening. I'll see you in the next video.