How Startups Can Crush Enterprise AI Sales | E2050

0:00a startup should be looking at like, well, what's different? And it's like, well, part of what's differing is we have reasoning as a utility. Think of every role in the economy as a bundle of tasks and those tasks, some of them fall to AI and some of them fall to humans. So I do think what startups should be looking at is like, how can I change the way, like if you look at workflow, the way that it's been built over the last 10 years, it's been built with humans at the center and that's the actual workflow. So if you were to zoom all the way out and say, oh, okay, well, I'm going to have a workflow that's going to be shared.

0:36Some of it's going to be through an algorithm and some of it's going to be through a human. What would that allow me to do? This Week in Startups is brought to you by LinkedIn Ads. To redeem a$100 LinkedIn ad credit and launch your first campaign, go to linkedin.com slash thisweekinstartups. Zendesk. The best customer experiences are built with Zendesk. Qualifying startups can join their startup program and get Zendesk products for free for six months. Visit zendesk.com slash twist today to get started. And Beehive, power your newsletters with AI tools, referral programs, and ad network features all in one platform.

1:15Get 30 days free and 20 % off your first three months at beehive.com slash twist. Hey, everybody. Welcome back to This Week in Startups. This is Alex, and I have a special show for you today. There's a theme going around the world of startups, which is AI hype, AI fundraising, AI applications, AI agents, AI chatbots, AI job loss, AI this, AI that. What I've always been very curious about is behind the headlines, how is enterprise generative AI spend going? And thankfully for me, a venture capital firm that I've known for a long time, Menlo dropped a very long and detailed report digging into precisely that.

1:53How is generative AI doing in the enterprise. So I decided to have a couple of the authors of that report come on the show to answer my questions. And then in a stroke of fortuitous luck, Menlo is perhaps best known in the AI game for being one of the backers of Anthropic. And this morning, the company announced that it raised$4 billion more from Amazon. So now we have the VCs from the report trapped under our thumb, so we can hit them with all sorts of questions as we would like. But just to welcome into the show. First of all, we have Joff Redfern. Joff, how are you? I'm fantastic, Alex. Thanks for having me on the show.

2:27Absolutely. And what I like the most about you, apart from the fact that you're part of this report, is that you were at Yahoo back from 2003 to 2009, right? I was. I was. I was early Yahoo prior to... I've been a product leader my entire career. Prior to joining Benlo about a year ago, I was the chief product officer at Atlassian. I was there for six years. And then before that, I was an early vice president of product at LivingDin when it was a tiny company, pre-IPO, and then helped grow that up into 10 ,000 employees. So I was there for seven years. Yeah. I love building things, started companies, sold a company, helped take a company public.

3:10That's a bit about me. Yeah. Yeah. But the coolest thing is that you worked at Yahoo where I've also worked. and it's really yeah it's like of all the things Alex that we can point out it's like I don't know if Yahoo was the one but I yeah I know Yahoo in the early days and you'll remember this was just super fun um it was like uh it was really the google of the day as the internet was emerging back in uh in the late 90s so it's it's always the tricky one because I I think you know younger folks and uh that are listening won't really know yahoo as the the amazing company that that we were part of in the early days it'd be uh like if an amazing institute like uh harvard was all of a sudden you know not harvard and it was at a different tier so yeah it's changed quite a bit over the years it's like if harvard became san jose state was purchased by private equity and i was gonna i was gonna go with katecott community college but katecott community college um yeah that's a deep cut.

4:12I'm in Rhode Island, which is why all Northeast jokes work. But we also have Derek Shaw with us. Derek, hey, you, my friend, I was prepping for the show. And the thing about your background that I love the most is that you were president of the Harvard Crimson. That's right. I started in journalism. I know. A lot of overlap, I think, with venture capital, but trying to, yeah, it's an interesting start. And a lot of admire a lot of the work that you do and excited to be on. I appreciate that. So also, Derek, on your Menlo page, you are tagged in on the Anthropic side of things. Now, I know it's Matt Murphy, who's the lead partner over at Menlo for the Anthropic relationship.

4:50But today, the company announced that it raised$4 billion more from Amazon, bringing it up to a total of$8 billion. And I guess the question that I want to know is why is it still so expensive for these AI companies? Because that's a lot more money, not that far afterwards. And to me, sitting here where I am, apart from the numbers and so forth, it feels a little terrifying. But I was hoping you could assuage my, oh my God, at that number. Yeah, I mean, I think Anthropic has a special relationship with Amazon. I feel like they have a very close partnership. And this is just a furtherance of that.

5:24But the other side of the coin is that Anthropic is one of the foundation models, right? This is when we made our first investment back in 2023. the thesis was that this was going to be one of the you know companies that will matter the most in the ai revolution and we've seen that play out that's why we have been getting closer and closer to the company and you know the announcement today it's probably just a furtherance of that and deepening of the relationship with amazon so joff you guys took part in the series c and then if i recall correctly led the series d is that right yeah we uh through an spv we were lead on 500 million of the billion dollar raise that happened on the d side and i take it now feeling pretty smart given how ai has gone since that deal was put together yeah well we're uh we're excited to share with you some of the findings that we've we've learned on the llm side uh as it pertains to the enterprise right there's obviously two large markets for llms we have the consumer-based market and then we have the enterprise-based market and it it's um you know it's starting to shake out that these are different markets and and how competition is approaching and attacking those are quite different so very very happy to see some of the data uh behind anthropics progress on the on the uh on the enterprise side i know that i'm slightly harassing you with the anthropic news when i asked you to come on to talk about the actual enterprise ai report but i'm just curious one more question about that does having a company like anthropic in the broader you know family portfolio does that really bring in a lot of information to the investment team that you guys can then i don't know learn from and apply directly to making new investment decisions or is that information segregated from the firm and so you guys can't go fishing in that pond to get learnings for we are we are very um you know church and state when it comes to that we're where Anthropik and both Menlo are, you know, we're not in there deep looking at people's usage metrics or anything like that.

7:31But I will say one benefit of the many that comes from it is just being able to see what actually is coming down the pipeline when it comes to new models. We forged a relationship, a special relationship with Anthropik five months ago. We announced our Anthology Fund. That's a$100 billion fund. And that's really looking for who are some of the most pioneering AI founders out there. And being part of that fund gives founders a number of benefit, early access to models,$25 ,000 in anthropic credits, access to some of the DevRel teams over there and some of the expertise. You'd be part of a network of fellow founders and builders in the AI space.

8:19And then every so often, once or twice a year, we run a builder. Say our first builder day with Anthropic was the 1st of November. And that was just, that was super fun. So it was at the Anthropic offices. We had a bunch of the experts from Anthropic coming out, meeting with companies that were building on the platform. And yeah, so we were able to have a really successful builder day. So I would say, you know, that part of the relationship is really helpful for both of us. So we're, we're, we really cherish the, uh, the team over there and they're, they're just amazing. I mean, the, the, the team and the, the, the caliber of the talent that has been brought to Anthropic.

9:03I just, I, I admire every day. It's, it's really quite talented. Well, with$4 billion more, they can certainly keep hiring. And I know one of your predictions in the report was a continuance of the AI talent drought. So we'll get to that in a minute. Let's start with what everyone wants to know, which is the high level numbers. So you guys wrote in this 2024, the state of generative AI in the enterprise, that AI spending surged to 13.8 billion this year, up 6x from 2023. Now, before we get into categories, Derek, Did that match your expectations? Is it a faster pace of growth than anticipated?

9:39To me, big number, big jump. No idea if it was bullish or bearish compared to your projections. Yeah, I mean, I think that when we did this report last year, we thought that this would take a little while to ramp up. And that's what you saw in previous technological transformations like this, right? You benchmark to cloud, you benchmark to mobile. And these are all trillion dollar kind of markets now that took a little while to get started. And so I think when one of our predictions from last year was that this would similarly take a little longer to ramp up despite all the spending that's going to it.

10:15Have you built something awesome? Are you ready to bring it to market? Well, that's great. But you're probably asking yourself, how do I get to my target customers? Well, here's an easy solution for you. It's called LinkedIn ads. And with LinkedIn, you can precisely target the professionals most likely to care about your product. You can use LinkedIn to reach people by job title, industry, company, location, and more. You know, all the stuff that you have on your profile page. Imagine you're looking to build your B2B audience. Maybe you want mid-sized companies. Maybe you want large ones. Maybe you want emerging small businesses, right?

10:45Small and medium-sized businesses, SMBs. Well, you got to build those key relationships and you want to drive results. Well, you just do that with LinkedIn ads and you're going to gain access to over a billion members of LinkedIn, of which 13%, 130 million of them are decision makers. That means they can pull the trigger on buying your product or service. And 10 million of them, 1 % of the LinkedIn audience are those elite C-level executives. You know what I'm talking about? Chief technology officer, chief strategy officer, chief revenue officer, chief executive officer, chief financial officer.

11:19They're all there on LinkedIn, and you're going to get two to five times the return on your ad spend. 79 % of B2B marketers say LinkedIn delivers the best paid media results for a reason. So here's your call to action. Start converting your B2B audience into high quality leads today. And we're going to give you$100 credit to start your next campaign. LinkedIn.com slash This Week in Startups to claim that credit. That's LinkedIn.com slash This Week in Startups. Terms and conditions do apply. I think what we've seen this year, actually is a surprise in terms of how fast that the revenue is actually materializing.

11:55And so you look at areas like healthcare or legal, which are actually traditional laggards in terms of technological adoption. And these are actually the verticals that are leading the AI revolution in many ways on the app layer. There's, you know,$100 million plus companies now in both of these verticals. And I think that that part was surprising is just like how much that app layer has really taken off. The reason why I'm actually less shocked by that than I thought I would be is that my spouse works in medicine. So I'm actually viscerally aware of that world and its inefficiencies. But if there are places where you could apply AI to a voluminous amount of written information, legal and healthcare are got to be two of the ripest apples on that tree.

12:40So even if they are historical laggards, doesn't that actually mean that they have more accumulated equivalent of like technical debt in their operations? And therefore, they're the best place to deploy. So actually, I can kind of see that, Derek. I guess I'm less surprised by that than I thought that I would be. Yeah, absolutely. And the other way that we look at it, too, is the comparison between the size of the total market versus the size of technology spent there. So if you think about somewhere, yeah, I mean, this goes towards, you know, the Silicon Valley kind of phrase now is services as software.

13:11And I think that it rings true. It's a cliche because of a reason, right? Because these are massive markets that tech used to not be able to touch. And now with AI, they can now automate a lot of that. You know, one thing that does surprise me and going back to this comment that Derek made is just the steepness of the curve on AI, right? Having lived through a number of these different waves over the years, you know, the internet came and then cloud computing came and then mobile came, they tend to be much slower. You lose fact that at the enterprise level, it was really, this is the Gen AI movement started two years ago.

13:46And I would peg that at January, 2023. It was January, 2023, that ChatGPT came out and said, hey, we got to 100 million mile faster than any other application out there. But it was a really telling story. I was still at Atlassian at that point. I was the chief product officer. And in January, that first earnings call, there was one mention of AI. But if you looked at all of 2022, there wasn't a single mention of generative AI. So it went from the first earnings call by the second earning call, there were 18 mentions in the transcript. So like a fun activity is go grab any company, right? And pull all of the transcripts from 2022 to now and load them up into your favorite LLM and just say, build me a chart of the number of mentions on AI or artificial intelligence.

14:45And what you see is it's like, it goes like crickets to, hey, something's happening here to all of a sudden that is the conversation. Yes. So I would say that is definitely one of the things that played out. And that's one of the findings that we have in the report. If you look at 2023, that would be the year really of the pilot. So the CEO gets off the earnings call with the CFO, goes over to the R &D group and says to the CTO and the CPO, like, hey, what are we doing with generative AI? And very quickly, what happens is teams get pulled together. It's like, how are we going to use this new technology?

15:26And that's really one of the findings from the survey is that last year was a lot about experimentation. This year, it's really about moving stuff into production. And that's where we get the 6x increase in the overall spend at the enterprise level coming up to close to$14 billion by our marks. I want to talk about basically data sourcing for the stuff because I love this particular chart. But to me, when I see this, I go, how confident are you guys in these numbers? Because you could easily categorize things in different pockets. I presume there's some bleed between them. And also, it's a growing industry.

16:06So can you just talk me through how this chart was put together? And if you're listening to the audio, this shows year-over-year changes in generative AI spend for foundation models, training and employment, data, vertical, departmental, and horizontal. Let me start with a high level and then I'm going to let Derek talk through how we've actually calculated numbers. In this chart, really there's two big buckets that you should be focusing on. One is around the LLM and the infrastructure needed to bring AI into the organization. And then the second bucket, these four, or actually the three bars on the right We're really talking about the application layer.

16:47So what we can see is that two-thirds of the spend that goes on at the enterprise,$9.2-ish billion, is sitting in the LLMs in the infrastructure bucket. And for those that can't see, by far the largest spend inside of the enterprise is against the foundation models themselves. $6.5 billion of that is spent there. So two-thirds is happening at the infrastructure layer. And then we have another third, which is being spent at the application layer. The main thing about the application layer, the big story is there is that that's an 8x increase from the prior year. So that's sitting at 4.6 billion.

17:28Then what Alex is asking, one of the questions he's asking is like, you can categorize your application layers a lot of different ways. In this case, we've broken it down into vertical, departmental and horizontal AI. And certainly we can squabble about like what belongs in the what bucket. But I would say the higher order story is that applications are coming, they're alive and well. And then for a variety of reasons, we've chosen to break it out this way. In terms of data sources, we went out and asked 600 IT decision makers. So these are budget owners that have purview over their organization's generative AI spend.

18:09This is a very lengthy survey, but basically, how much are you spending on generative AI? What is that relative to your overall spend? And then very specifically, what tools are you spending it on? And I think that one of the motivations behind the survey was that there's just a lack of good data out there of like, what are actually IT decision makers really kind of looking at? What are the use cases that they have? And where is their money going towards? And so this is, you know, we spoke to 600. We didn't get to all, you know, the Fortune 2000. And so there's obviously, you know, this is directionally our answer.

18:43um but i think that it's pretty informative to see some of the insights of like what folks are actually you know not just playing around with anymore but actually adopting so just to summarize that um the 4.6 billion that we see for the different types of ai applications you guys surveyed on i should probably pay a little bit more attention to the year of your growth in that number than to sit here and go why is it 4.6 and not 4.7 billion yeah exactly and then the other thing, the next question you're going to ask, I was like, okay, great. What are they doing at that application layer? And the thing that I would say is fascinating about that is that there's a real broad set of Gen A use cases in the enterprise.

19:25So sometimes you see things get very concentrated around one department or one use case, but I think it speaks to the usefulness of the technology. So when we break down if on average an enterprise has 10 gen i uh use cases amongst it which ones are most popular yeah and that would be a that'd be an obvious question so when we look at it code generation was number one 51 percent of the folks on the survey uh said that they're using ai for code gen they're building software for it joff just to be clear not 51 percent of the 600 who are paying for generative ai services but of all the 600 51 companies are paying for um code completion tools yes okay just wanted to make sure so this is as big of a number of the 600 as we could imagine okay i appreciate that keep going that's not surprising we've seen um github uh copilot is one of the you know fastest growing revenue products out there and in the application layer of AI, they're north of 300 million in annual revenue.

20:32We've seen players like Cognition, Codium, All Hands come about. After CodeGen is support chatbots. Probably not surprising there, 31 % are saying they're deploying chatbots. We've got products like Sierra, Decagon, Acero, which is from an ITSM, IT service management use case is a big one there. Hey, startups. When you're a business, you got to treat your customers right. Unreasonable hospitality is the standard today, but you're going to need tools. You're going to need a platform to help you do this. And that platform is the Zendesk Suite. The Zendesk Suite is going to give your startup all the tools you need to deliver exceptional customer experiences so you can build stronger relationships without growing your headcount.

21:22That's key, right? Every dollar matters. You got to control headcount. You got to control spend. So use the tool that Shopify, Squarespace, Uber, and Instacart all rely on. It's called Zendesk. Let's take a look at another customer, Unity. Very famous company. They saved$1.3 million with Zendesk automations and self-service, and they saw an 83 % increase in their first response time. These companies love Zendesk because it's so easy to set up and it scales with you as you grow. They'll also give you all the metrics to make your reporting easy, keeping you and your business agile and investor ready.

21:58And that's because you're their esteemed customer. And they've created the Zendesk for Startups program just for you, where you get unlimited access to all the Zendesk products, expert insights, all the best practices, and entry into their amazing community of founders, all at no cost for the first six months. That's right. They want to support you. zendesk.com slash twist. Get ready to scale with the best in customer support with six months free, nothing to lose. Then you move to enterprise search and retrieval, 28%. These products, a product like Glean has been around for a while, but with the emergence of Gen AI, what they've been able to actually accomplish on the search front has been dramatically better than maybe what they were able to do in their early years.

22:44Another product in there would be something like sauna yeah then we move to um you know things like uh meaning summarization uh you know so many of us are remote workers and we're so used to seeing that ai agent has been added to either pull a transcript or summarization of our conversation yeah um things like firefly and and otter and then the last one i would mention it kind of round out the top five would be um would be copywriting and this goes back to the comment that you were making out it's like i you know lms are like calculators for words they're like really good at authoring content so i you know you could certainly see a world in which everyone um would be using uh or having a copywriter or an editor you know grammar checker just part of whatever they're communicating to help them be more precise and concise products like writer typeface where we have an investment copy ai they tend to be you know some of the leaders in that space but that gives you a feel for like i could get what is going on at the application layer of the enterprise so a lot of words is my read of that i think the calculator for words is a great way to think about this cogeneration is creating characters on a screen support chatbots create characters on a screen you know meeting summarization characters on the screen copywriting characters on a screen the thing that surprised me is that workflow automation was so low in this chart to date because if you go back five years we're thinking about ui path and you know rpa robotic process automation and there was a lot of enthusiasm pre-generative ai pre-llms that rpa was going to remove a lot of the human drudgery from digital work.

24:33Then we got much better tooling. But now that we're looking back at 2024, I'm not seeing as much of that show up as I kind of expected. And this is a long way of saying, how much progress have we actually made on agents, I suppose, that can take some of the work from us and do it versus being more assistive in our day-to-day work. And Derek, I see you blinking at me, so go ahead. Yeah, I think that we think about it in terms of waves. And so the first waves of Gen AI apps were what we call RAG apps, or retrieval augmented generation based. And so usually, you have an external knowledge store, and you use it for things like synthesis.

25:14So Eve is one of our portfolio companies. It's a legal co-pilot. And what it does is it takes a lot of long, dense legal texts and makes it generates reports, it generates legal briefings, things like that, so that the lawyer on the other and doesn't have to go through the drudgery of all that work. And so that's kind of the first generation. And then as you get more advanced, we move into agentic architectures, which, you know, today our survey found that it is a minority. But if you ask a year ago, if you look at our previous report, it didn't exist a year ago. And this idea of you had like baby AGI and Autogen and some of these like, you know, open source projects back then, but it didn't exist as an enterprise idea of something that, you know, when I think about traditional RPA like UI path, the idea of applying it to the enterprise hadn't really existed.

26:03And now it does. So Derek, it sounds like what I was doing was just being impatient. And now it is showing up. And so what I expected to happen is I just had my timelines off mentally compared to the market. I think the technology is now there. And if you look at things that we're really excited for in 2025, this is one of the things that we think will explode is moving from retrieval-based architectures to more agents and things that can automate workflows across horizontal areas. So like you think UiPath, but also verticals, you know, healthcare, there's solutions like Tenor that are doing kind of ingest automation and, you know, a lot of different domain-specific applications, I think you'll also see.

26:44And I'm literally right now pulling up the OpenAI 01 preview blog post because I forgot the term that I need here. it's like time series thinking when models take a little more time before they make a decision or return a prompt is that underpinning the improvement in the technology that is making the agentic approach more feasible today derek yeah um test time inference and so i think that i i think that there's a couple layers to this right um so you can have stuff like oh one which is uh kind of formalizing a designed um uh kind of pattern at the model layer and so as the model gets smarter, it makes fewer errors.

27:21I think the problem with agents that you had traditionally is that it's kind of run in a recursive loop. So if you think about it, the agent, which is the LLM, will think, okay, in order to accomplish this task that requires 10 steps, what are those 10 steps? And then I'll go out and do it. And then I'll think, I'm at step one. Okay, check. What is step two? And then if you think about error rates there, LLMs hallucinate, that's now well established. If you have a 99 % error rate or a 99 % accuracy rate in step one, if you haven't that for all 10 steps, by the time you get to step 10, the error rate is something that is unacceptable for enterprises.

28:00And so that's traditionally been the problem. And so LLM's getting smarter is part of the answer. But also when you apply it to specific domains, you have data scaffolding around it. We like to use the term agent on Rails, which is basically you need to hard code it or harness kind of the domain of like all the actions that the agent can take you need to kind of set guardrails on it with hard with code in order to point it and get you know higher levels of accuracy so that that recursive error rate doesn't compound too much do those guardrails have to be programmed on a per use case per industry or per company basis i'm trying to figure out how hard it is to hard code those because to me that could be incredibly complicated or relatively easy i just don't know where it lands yeah i think it um people are trying to figure that out right um and i think that obviously the more specific to a particular use case the higher accuracy and more robust it is and but the thing you're trading off really is degrees of freedom which like you know all the way at this very end is agi you know no guardrails just the model just like put it in a for loop and it runs and then on the other end is um what we have today which is computers that are 100 % hard-coded application logic is determined by uh the computer or by you know first of all some programmer who sat down and was like okay i'm thinking in the shoes of the user what do i need and so i think the answer will probably be somewhere in between we're trying to move towards agi eventually but yeah i think right today we need uh stricter guardrails i think you know there's um as i'm hearing you talk to eric there's There's a good example in the software agentic space, right?

29:39Okay, so let's take code generation. And we can say, well, how much progress are we making in code generation? One way we can look at progress is we can look at this score called the SWE bench, right? And we can look at that over the last what's happened there in the last 11 months. So SWE bench, for those that don't know, it's testing real world tasks faced by software developers. And the benchmark was based on things like pull requests and issues from open source GitHub repositories. And I think there's something like 2200 tests in there. So if we go back to January of this year, about 4 % of those tests were completed by the best software agentic system out there.

30:22In March, there was a company called cognition, which got a lot of traction, you might remember that. By March, it was 14%. Now, if you look at it, the number one score on Sweebench belongs to one of our portfolio companies. It's an open source software company called All Hands, and they can solve 53 % of those cases. Hey, founders, let's talk about a super weapon that your startup might be missing, killer newsletters. Newsletters aren't just a great way for you to stay relevant. They're also an incredible growth engine for your startup. social platforms and search traffic, those things are losing their edge.