E62: Wing's Chris Zeoli on AI Infrastructure Opportunities

0:04Welcome back to Turpentine VC, the podcast where we discuss the art and science of building successful venture firms, VC to VC. Today, we're joined by Chris Zioli, partner at Wing Capital, who shares Wing's data-centric investment approach as early backers of companies like Snowflake and Pinecone. We discuss Wing's enterprise-focused AI thesis and break down the competitive dynamics between OpenAI, Meta, and Anthropic. Chris also offers insights into the massive infrastructure build-out reshaping data centers and semiconductors. Let's dive in.

0:39Chris, welcome to the podcast. Thanks for joining. Great to see you, Eric. Long friend. Yeah, likewise. So Chris, let's start with a brief introduction in terms of what is the scope of things that you invest in over the last few months, last year? How have you narrowed down the categories that you're particularly interested in investing at the moment? Yeah, totally. So I have been at Wing for a little over two years. We're an enterprise focused investing firm, so we don't do any consumer, don't do any international or fintech. We're very focused on kind of how data is being put to use in the modern enterprise.

1:12So that leverages a few different key components. We were early backers of Snowflake, Cohesity. So kind of building off of that foundation of data infrastructure within the enterprise. And now kind of the last focus of the last two years, after about 10 years of focusing on unstructured data, is taking on all these unstructured data challenges. And that's kind of where AI comes into play. So that's all these new data sources that have really not been tapped. So that's the macro theme. The micro theme within the macro theme is a lot of different infrastructure vendors. So happy to dive into that, too.

1:49Yeah, but first, let's even start more high level, which is just AI investing. How do you identify the different areas that one can invest in? And how do you and your firm think about, okay, these are areas we want to play in. These are areas we don't want to play in. And why is that? Yeah, so we borrow a lot of thinking from the prepared mind framework that Peter learned at Excel. So we're spending time within a few core focus areas that we're really highly interested in. We kind of have an outstanding belief that data is, and I've felt this for a long time, is that data is kind of the foundation of AI.

2:27So within that, we kind of mapped the flow of data from all these different sources and how you can actually put them to use. So we're spending time talking to buyers of these data systems, thinking about what are the most strategic sources of data for those enterprises. And then you kind of identified the data. Now it's kind of actually putting that into production and building the system that flows all the way from the source, whether that's emails or documents or all types of stuff like that, to the processing of that data, whether that's the embeddings model, where we have some investments that are actually Voyage AI, which is actually focused on training better embeddings models.

3:08So some of those are done. Some people use OpenAI's model. Some people kind of have turned to Voyage, which is higher performance for certain chunking of information. We're looking at RAG pipelines as well. So RAG has kind of become the dominant architecture that people incorporate their private enterprise knowledge into AI applications. that's all stored in a vector database where we're backers of pinecone the leading company behind the vector database architecture as well as how does that get put into production into an application whether that's a internal application or an external application most of the early usage of ai was internal applications much easier when you don't have to worry too much about the data and security governance issues.

3:54But when you want to expose, you know, a customer support bot or an insurance claims bot or something to the end customers, then a lot more risk comes into play. And that's kind of what creates a lot of the security opportunities within AI. But that's kind of the general flow. There's a lot of other parts to getting to a better model, whether it's like what used to be called fine tuning. now is kind of more morphed into RLHF and kind of the small model thesis, whether latency or domain expertise are kind of the two things that people are heavily focused on. So we're investors in gradient AI, and I'm on the board there.

4:34That's kind of one of the leading companies putting small models into production at financial services firms, tech companies, healthcare, et cetera. I want you to zoom out even further. You wrote a blog post called the talk about the major computing cycles. What did you find there? How did you think about why was it helpful to situate AI within these different revolutions? And what does that tell us? Yeah. So I learned about this. It's actually Jensen's framework for kind of the major data and computing cycles and how we got to here. So the whole internet is based off of this retrieval technology, which is basically you kind of make a query, something comes back on the other end.

5:19It's very structured. There's kind of not really much room for error. And it's all kind of one-to-one mapping. IP addresses are kind of a good way to illustrate kind of how retrieval architectures work because you have something directly to map something to. Jensen talked about, we're kind of moving into this new era of computing, which is retrieval plus generation. And that's where the retrieval augmented generation comes. Generation really augments that retrieval process, which is the core of how everything from like Google to file systems to websites work is like they're performing a retrieval oriented task.

5:58So he kind of paints this picture of generation is kind of the future of all computing, whether it's internet-based or knowledge-oriented. That kind of is a completely different architecture to actually run that generation process. You need totally different hardware. It's super demanding. You can do multiple levels of generation. I mean, there's kind of more simple oriented generation. Like you ask a chat GPT a pretty simple task and it spits it back. Like, let's say asking it about who the first president of the U.S. is. There's not really that much generation that needs to go behind that. But if you ask it something much more complex where it needs thinking and reasoning, that's where generation really comes into play.

6:44And that's kind of how he describes the next architecture cycle that we're moving into. too. That's got it. That's helpful. So if you break out the investing world into the models on one side, you guys don't don't haven't made any bets there. You think it's kind of too late there. Well, it's a it's an interesting one. I mean, we have looked at a lot of vertical specific models and we have a bunch of companies, particularly in the drug discovery space that are doing that are training their own models. So we do believe there are opportunities for models in specific data realms or specific verticals that have some unique aspect to it.

7:25I'm not an expert on the drug discovery side of things. My partner, Sarah, Ansu, and Garavar, but they're basically trying to predict how proteins will react to different molecules. And that's kind of the foundation of how they're thinking about their models there. I mean, in the large language model universe and then multimodal, I'll kind of get into multimodal too. In the large language model universe, we do think it's super capital intensive. So there's kind of an ongoing battle between the closed source players, OpenAI, Anthropic, et cetera. And then the open source players, some of which are open, but are kind of backed by massive entities like Meta that are pumping in$20 billion just this year into the chips to actually train them.

8:12Mistral is kind of the next closest open vendor, although they've kind of pulled back some of their open models and are quite, so you kind of have to be, you kind of have to raise at least a billion dollars to play in that world. It's interesting. We kind of see we're getting to a point of diminishing marginal return in models, which like everything is kind of getting to this roughly 90 % accuracy benchmark and kind of each subsequent round of training, you're getting less and less incremental performance. So we have a very active discussion of, you know, are we moving towards a world of model commoditization or are we moving towards a world where kind of the leading model providers that are accumulating more and more data from network effects and people typing in prompts, getting answers, people rating answers.

9:05It's an open end of discussion. I tend to believe we are moving towards model commoditization, though, and distribution is going to become more and more important. So that's how we're thinking about the broader landscape. What do you think Meta's most recent move means for OpenAI? Well, it means they have a competitor who is as well-funded as them. So they're going to invest the same amount per year. I think the information just put out today that OpenAI will spend$7 billion this year on inference and training. Their total staff costs are$1.5 billion. so it's it's pretty insane they're spending five times more on on computing than they are on on actual people meta i mean they're they're 20 we're gonna invest 20 billion this year so they are incredibly well funded they will probably be able to compete on multimodal stuff i expect them to invest a lot they're just given the nature of the the facebook family of apps being image-based being video-based, media-oriented, they're not going to want to lose that battle.

10:17So they're going to invest on both of those fronts. I think OpenAI has an advantage with the user feedback particularly and collecting all this massive amounts of user data. That was a huge boon for Google as well. Obviously, PageRank and the user behavioral data was really what put them over the top between Yahoo and others. It's kind of widely stated that in the early days of Google, the accuracy was not that much better. It was pretty marginal versus other search engines. But that new technique of PageRank plus all the user behavioral data that they began to gather and incorporating that into the product was really kind of how they improved the accuracy of the retrieval process that they kind of built that we all know is now search engines.

11:13I think that's going to come into play too with OpenAI. And like, you'll get more and more of those thumbs up, thumbs down, which of these answers do you like type prompting and the outputs that the model gives you. So there's going to be a data network effects game over time. And I use meta models sometimes, but I find myself going to them less just because of the way that they're productized. So I think that that will become increasingly important. So you're bullish on OpenAI long-term? I am, yeah. I mean, my colleague Zach just put out a really good piece about the business model of OpenAI and them making 55 % of their revenue from consumer subscriptions.

11:57I mean, they have about 8 million global consumer subscribers. I don't see why like all of the, all the world's knowledge workers won't, won't have at least one of these models. And I think they're going to get better and better at understanding the users that they have. Like you can kind of really do deeper research on the type of queries that they're running. Let's say I'm asking a lot of queries about data infrastructure, cybersecurity, those type of things. Like it kind of gives them more, more, more kind of awareness of what type of datasets they need to be going after, how they need to be improving their models for certain types of users.

12:37So I think they have a big, big advantage there. I mean, the API business that they have also, I think is very strong. Their developer interface is definitely the best. We've seen a few of our portfolio companies generate huge amounts of interest just from being in the docks of OpenAI. And that's kind of become a powerful place to be as a startup. Some of the most, the highest street cred that you can gain is getting endorsed in their docks. So I do think that they have a really big lead. I think the consumer side and the enterprise side will be different. On the consumer side, it'll be Gemini, who's probably going to become the closest challenger for a lot of the type of queries that get routed to ChatGPT.

13:23And on the enterprise side, which is like 45 % of their business, I think that's where they'll see Anthropic more. But I mean, they have such a big lead there, especially coupled with Microsoft and some of their cloud partnerships. That'll be hard, but there's kind of like a central versus access powers type dynamic forming with Microsoft and open AI and then Anthropic, Google and Amazon, who are both the two biggest investors in Anthropic. So I wouldn't underestimate the weight that Amazon particularly will have behind throwing itself behind Anthropic. I think Google is going to be really focused on the consumer side and kind of trying to compete with ChatGPT.

14:06And let's see if they can catch up. They obviously were stumbling earlier this year, but are looking more formidable now. So I kind of break it down into two sides, the consumer side and the enterprise side. Say more about Meta specifically. Do you think this is a mistake? Or like, what is sort of the decision that they're, or sort of the strategic thought process that they're having, investing$20 billion a year into this? And are they prepared to just keep scaling it up? Or where does this play out for them? Yeah, I mean, Zuck certainly goes big. I mean, he was quoted today saying, you know, I'd rather over invest than under invest, which I certainly respect.

14:44I mean, he's, he's, he's behind the biggest platform and consumer over the last 20 years. So he's playing big. He's obviously made a big bet with Oculus investing 50, 15 billion a year there. So I think he, he doesn't want to be beholden. He, Facebook runs by far the most number of inference calls of any company globally. I believe they run a trillion inference workloads a day, which is over 10X, the next closest. So they don't want to be beholden to anybody who's charging a variable cost on models. Makes sense. They are very much beholden to this chip providers though in training this for themselves.

15:25So about 14 billion of that 20 billion that they allocated this year is just going right to NVIDIA. Pretty massive, over 10 % of NVIDIA's revenue. We'll see what the upgrade cycles of these data centers look like. I mean, that's a huge topic of debate is, you know, will people need to be getting the latest chip every year or will these be like three to five year or 10 year upgrade cycles? People don't really know yet. A lot of debt financing is happening on the back of really long term upgrade cycles for companies like CoreWeave and Lambda. And so that's a really important variable. It's like how long will these actually last?

16:08Are you going to get a few good training runs? Are you going to get the next two families of Llama and then have to upgrade? Or is this going to be a really long-term investment that doesn't depreciate super fast? It is moving to one-year upgrade cycles, kind of like Apple did in iPhone, which will put pressure on competitors like AMD, who are, you know, they released a new chip. It's slightly better than NVIDIA for a few months. And then particularly during the last kind of last three to six months of the prior gen. And then NVIDIA launches new one. And NVIDIA is trying to close that gap basically by going to one year upgrade cycles.

16:50So I think it's bold. I mean, it's economically irrational, at least to this point. Like they're not charging for inference. They're not charging for consumer subscription. They're not charging for APIs, for access to their models. It's more, we don't want to be relying on other people's infrastructure at all. And that's kind of an engineer's mentality in Silicon Valley. Sometimes you end up spending$20 billion on something that you could have just bought from OpenAI for a billion. Other times, it becomes super important and you kind of end up building the next, that's kind of how AWS got built is, is they didn't want to be beholden on other people's file infrastructure.

17:32So it can lead to huge innovations. It can also lead to big R and D efforts that don't work sometimes. Yeah. That's well articulated. Hey, we'll continue our interview in a moment after a word from our sponsors. How deep do you go to seek out an answer to a question? Maybe you've spent hours clicking the source links on an obscure Wikipedia page. Or maybe you're even the type of person who checked out the entire shelf on the topic at your library. If you're nodding along, then check out GiveWell, an organization that researches questions about global health and philanthropy, even if a satisfying answer might require years of reviewing studies, talking to experts, and over 300 footnotes.

18:12GiveWell has now spent over 17 years researching charitable organizations and only directs funding to a few of the highest impact opportunities they've found. Over 125 ,000 donors have used GiveWell to donate more than$2 billion. Rigorous evidence suggests that these donations will save over 200 ,000 lives and improve the lives of millions more. GiveWell wants as many donors as possible to make informed decisions about high-impact giving. You can find all of their research and recommendations on their site for free. You can make tax-deductible donations to their recommended funds or charities.

18:43And GiveWell doesn't take a cut. If you've never used GiveWell to donate, you can have your donation matched up to$100 before the end of the year, or as long as matching funds last. To claim your match, go to GiveWell.org and pick podcast and enter Econ 102 with Noah Smith and Eric Torenberg at checkout. Make sure they know that you heard about GiveWell from Econ 102 with Noah Smith and Eric Torenberg to get your donation matched. Again, that's GiveWell.org to donate or find out more. So, okay, we talked about foundation model providers. We'll get more into sort of the infrastructure stuff, which you're very active in.

19:19How about on the application side? How do you think about that? And are you doing a bunch there? Yeah. So I'm not as deep as my colleagues Tanay and Zach there, but we have done a lot of investing there broadly. We're very intrigued with a lot of these vertical vendors, particularly in markets that have lots of unstructured data. So companies like Harvey and Legal Space, we look to with a lot of respect. We look at Glean as a company that is bringing kind of rag-like approaches to the application layer. We're looking for other new verticals as well across a bunch of different applications, but I spend a little bit less of my time there.

20:01That makes sense. Okay, so let's focus on the infrastructure side. You started to a little bit earlier, but maybe you can sort of map out even further the different kinds of opportunities and where you think they're best opportunities versus where maybe they've already been picked over or it's unlikely to be venture. Yeah, totally. It's a hard problem. I mean, OpenAI is trying to compete across different parts of the stack and get more into the data flow. The data side of things is really where we see the most opportunities. That can be parts on the ingestion side, helping you handle particularly these newer data sources that haven't historically lived in warehouses or databases.

20:44That's where the new data and the incremental performance is going to come from. So we like all the tools to help you build RAG pipelines. So that can be embedding generation, which we've made with Voyage AI. that could be extraction of information and entity mapping. So for example, in a term sheet, the board is not like a wooden board, but like an LLM would interpret it as. It has to be mapped to an entity. So there's a lot of that his or hers in a text document. There's a lot of those really complex technical problems that kind of map the search in a lot of ways. Still a really unsolved problem that we think is going to be increasingly important for LLMs to get to that last kind of, we're kind of at this phase where we're getting that last 10%.

21:36How do we go from 90 % to 100 % accuracy? We'll probably never get to 100 % accuracy, but how do we get incremental performance from here? Those are a lot of the techniques that are coming into play, particularly on the data side. We're very interested in all this kind of structuring of both unstructured and then different modalities of information. So there's a lot going on on the images side of things. There's certain companies we're very interested in that are using models themselves to label certain types of data sets. So kind of going after the human labeler and things like image labeling, audio labeling, where we're investors in DeepGram AI, which is kind of the largest text-to-speech infrastructure provider, we're seeing data become increasingly important for all these infrastructure providers.

22:33And they're kind of thinking about how do we get the new sources of data in the cleanest way as possible? And how do we get as much enrichment of that data as possible? So we're really focused on tools that help drive that. Do you have a broader request for startups or things that don't exactly exist yet or that you haven't made bets on that you would really love to? Yeah, I'm very interested in this extraction-oriented product that can pull information out of different content types and structure it. There's kind of been some early ingestion tools. Some of the legacy ETL vendors have kind of bolted on some support, but they're not doing extraction and transformation.

23:21So that's my top request. I'm very interested in the logging and monitoring space as well. It's obviously a huge historical market, like Datadog, Splunk, AppDynamics. There's actually probably 10 more that are worth over$5 billion each. That was historically a space where there's information overload. It's all structured information. and it's there's a really important decision from the end user of like which which incidents and which logs do i investigate so i think models can incorporate a lot of unstructured behavioral data there and that is a space that i'm pretty optimistic will get reshaped i mean splunk just got acquired within cisco there's kind of a a lot of a lot of excitement about what models can do there that that's both like security oriented plus like application performance oriented so very interested in all that's going on in log land as well yeah the you've we also we touched briefly upon it on on data center stuff say more about how you how you think about your opportunities there how you approach that space yeah i mean it started from like the observation that you know an increasing percentage of companies are coming in asking for half of the round to go to GPUs.

24:47Thankfully, we've kind of moved past a lot of that, although we do see tons of companies spending multiple millions per year there. So it started with, you know, what's going to happen with this computing need? Are we going to be in a world where, you know, gross margins actually go down and computing just becomes increasingly costly for all the companies we want to invest in that want to incorporate AI. And then it began to trace into all these other challenges with this major data center upgrade cycle, whether that's networking and sharing data across huge GPU clusters. We're kind of in an unprecedented era where you have thousands of machines processing huge amounts of information together and they need to stream what they each have done to each other.

25:35So that's like a networking problem that is at a scale that is just, not not precedented before so so going into that learning about kind of some of the the ai networking initiatives that are going on with with some of these companies like arista networks and nvidia who acquired melanox kind of having an opinion there then even tracing it down to like what's going on in memory how does this data like actually get get transferred to the machines even even things as niches like kind of components in in the data centers so with companies like estera going public we've seen a couple of new companies emerge that are solving different parts there even have like looked at things like that are super niche like cooling where there's like both the industrial scale cooling of these massive data centers like building like hvac like systems to like, how do you actually cool the chip and prevent the chip from overheating?

26:36So like we've heard from a lot of GPU clouds and big data center operators that as much as 30 % of these chips are overheating every year and become unusable. So cooling is kind of like insurance on your huge GPU investment. It can also mean if just a few parts of your GPU cluster go down, like the whole system can go down. So there's kind of some reliability issues there with cooling that we've been interested in. There's so many things. I mean, a lot of talk from people like Sam Altman about things like power and building nuclear reactors and how we're going to run out of electricity. And I'm less of an expert on that, but it just shows the complexity of building out these data centers.

27:24So I've tried to go deep on all those different subcomponents and think about which ones actually could be viable for a new company. Let's also get into cyber briefly, because that's also a space you've got to do. But why don't you share your thoughts there? Yeah, I mean, cyber is in a moment of change. I mean, we know we need to get better models. The way you get better models is more data. The way you expose all your data to a model system creates a ton of security risk. So model developers and data engineers, they're tasked with building a better model. So they're exposing everything. The way you get the best model is literally not to hide anything.

28:09So it's kind of an unprecedented security risk. It's kind of like an ask for forgiveness type security strategy. So when these models, particularly the ones that incorporate private enterprise data, started to come out, the governance challenges were like people were aware of them right away. And, you know, there's some easy fixes like don't reveal salaries and don't reveal personal information and stuff. But there's a whole bunch of much more complex ways that really sensitive information can get exposed. And actually understanding what's sensitive and what's not is a huge problem as well. Security has been moving towards a world where it's not necessarily all top down driven from the CISO.

29:06in the way that it used to be in the era of kind of network and endpoint security. We kind of saw companies like Sneak emerge and GitHub emerge where security became in the hands of engineers and it was kind of an everybody responsibility. I think what's going on in model land is an extension of that trend that's already been happening where everybody who works with data is going to be responsible for security in some aspects. In some ways, that can broaden the market for security a lot. So if you have a SaaS company that's scanning what goes into models and you have a huge team that's trying to incorporate all the different data that they're working with into the models, you can see how that'd be a much bigger business than a business that just sold to the CISO.

29:58So that's what I'm excited about, is that it's becoming a more democratized use case where everybody kind of pitches in a little bit versus kind of one person sitting in a Fortune 2000 company and scanning down thousands of alerts and kind of thinking about what to focus on. I want to address some of your... You have this great blog that people should check out. You wrote this deep dive on the state of semiconductors at the end of last year. What do you think are some of the main things people should know about the overall semiconductor landscape and how to make sense of it? Yeah, yeah, it's a super good question.