AI Engineering Revolution: Winners, Chaos & What’s Next | FirstMark

0:00Cursor announced they're at$500 million of ARR. Lovable has gone$0 to$60 million of ARR in the last two quarters. GitHub Copilot,$400 million of ARR. There really is no plateau in sight for innovation. You might miss something spectacular and industry shifting. Welcome to the Matt Podcast. I'm Matt Turck from FirstMark. Today, we're doing something a little different. My FirstMark colleague David Waltcher and I sit down to unpack one of the key trends of the year, the rise of AI coding. We talk about key startups in the space, draw some lessons from history, and think through second-order effects on the rest of the ecosystem.

0:34With every surge in production, there's just a cleanup crew that naturally comes and a new market and industry that falls in its wake. If thanks to AI, everyone can now be a developer, then what does that mean for software, the role of the CTO, as well as the broad tech industry going forward? For CTOs, this is going to be sort of a hallmark moment for them over the next five years or so, where they have a lot of important decisions to make across talent, architecture, team structure, governance, principles, security. And what opportunities and challenges does that create for engineers, founders and investors?

1:06All of the problems that I'm talking about are very much our opportunities as investors and founders. And we see a ton of them across all of these spaces. I'll call out a few. This is a VC lens on one of the biggest stories of the year and very relevant for anyone trying to make sense of the ever-changing AI landscape. All right, David, excited to do this. Today, we're going to talk about a really hot topic, the impact of AI on engineering. So it's a little bit of a different format for this episode of the Matt podcast. We have a presentation. We're going to go through it and talk about it together.

1:42So I'll let you drive and get started. Yeah, well, thanks so much for having me on. So quick context, I was asked to pull this presentation together for the CTO summit that we threw last week. You know, I would say it is one of the most interesting times in engineering and this broader ecosystem of technology. And so maybe I'll just kick off the presentation with talking about a quick summary of how we've arrived to this generative AI moment in our engineering ecosystem by just giving a quick highlight of the six trends or so that have demarked the last two decades in the world of developers. So I'll start with pre-cloud when we used to just deploy code directly onto servers.

2:22A.k.a. prehistoric times. Correct. And then around, call it 2010, along with cloud, DevOps finally got coined, which was really the merging of dev and operations into this whole subsequent tool set and set of workflows that emerged around it, like Git, CICD, etc., that we all know today. And that very much spurred this generation of SaaS explosion. All of the small, mid, large cap companies today that we think of as being venture backed very much belong in that category of company. And then two sort of things between then and now that I think are worth noting. One is the proliferation of APIs, which essentially gave all of those SaaS applications the ability to very quickly expand their capabilities by almost outsourcing things like payments and messaging and search to companies like Stripe and Algolia and MessageBird.

3:17and then two is abstraction so you think about companies like HashiCorp and Sneak and dbt and Vercel very much catering to an audience of developers who are using code interfaces to all of a sudden do things that historically have been very much outside the purview of developer workflows and so think about infrastructure provisioning with terraform or think about dbt when all of a sudden we're now data engineering through a code interface and and over that time So from, call it, 2006 all the way to now, estimated about 7x growth in the number of global software developers, which I think of as sort of an index to this tech ecosystem.

3:58And that's where we've arrived today. So generative AI, this massive shift in delivery model, in the way that developers work, and one that I, you know, in this presentation really talk about being probably the most impactful thing on this page. It's interesting to think about why coding and development has been such a major success in generative AI. When people talk about, okay, what generative AI applications have been successful so far, coding seems to be number one by far. There is something about code that lends itself particularly well to AI. First of all, because there's tons of training data out there.

4:37So GitHub has been a godsend for AI training because so much of GitHub is public. So publicly available repositories. We're talking about hundreds of millions of repositories on GitHub. Two, the coding language, by definition, is highly structured and very precise. It's a grammar that you have to get right for the machine to work. And then three, it operates through patterns. which lend themselves well to AI training. And then four, that's a very clear ROI to all the things. So people spend a lot of time doing grunt work around coding. So anything that enables them to automate the work is incredibly impactful.

5:27Yeah, I think you nailed it. The first point you made is really interesting, which is this corpus of open source code that all of these models could use as training data, which is very much true. And actually, I think later in the presentation, we'll talk a little bit about some of the bad or somewhat negative effects of that corpus of data that those models have been trained on and how that's created specifically security concerns. The empirical nature of code has certainly made it a great early use case for AI. And then the last thing I'd say is just developers have always been the type of persona that loves to try new things.

6:05And so as a buyer and a user, they always made a ton of sense for AI. And obviously, you know, across the broader world of consumers, it's been well-documented that ChatGPT has made its way into, you know, millions of people's hands. So it's not to say that AI is not a mass market phenomenon already, but I think from an enterprise perspective, they've just been a great buyer for tools like this and the results have been very much measurable. Yeah, that's a really interesting thought. in that in some ways, generative AI has accelerated a behavior that already existed, meaning that developers have been using Stack Overflow for many years.

6:46So the idea of using somebody else's code or pre-existing code, copying and pasting, is not a new behavior versus a lot of other things that people outside of coding need to do with generative AI which is, you know, figure out how to use those things. In many ways, it's leveraging and existing behavior. And also, the concept of having code completion in the IDE is not new either. I believe IntelliSense was created in 96 or something like that by Microsoft, which was already an automatic code completion. So that may be another reason why adoption has been so dramatically fast. Yeah, absolutely.

7:32And so maybe to kind of exemplify a lot of what we've been talking about, we're about 24 months into this wave, give or take. It's changed a ton of behavior. It's taken our community by storm, but it's also just created a ton of really, really interesting special companies that have grown especially quickly. And so Cursor, maybe three weeks ago, announced they're at$500 million of ARR. Is that actually the fastest growing company of all time, maybe in B2B? I believe so. Lovable, which allows people to create prototypes and web apps through a series of prompts, has gone zero to$60 million of ARR in the last two quarters.

8:12GitHub Copilot, which I know you just had Thomas on the podcast not too long ago. $400 million of ARR, 15 million developers, very much the steward of this category, first mover. Yeah, exactly. The interesting thing that Thomas highlighted, among many other things, and this is really an episode worth listening to, was how they actually were thinking about AI when GitHub was acquired in 2018. So it's not just that they released Copilot a full year before the whole ChatGPT craze, but they had been thinking about it way ahead of time. I thought that was really interesting. And yeah, Copilot came out in 2021 running on Codex, which was the first coding model by OpenAI, which came out in 2021, I think, which was basically based on GPT-3 at the time.

9:09And recently has had a large resurgence in a new form. A couple other examples. So V0, again, you just had Guillermo on the podcast. Another great episode. Sorry, I'll stop shilling the podcast on the podcast, but that was phenomenal. He's such an incredible founder. Winsurf, which is rumored to have sold to OpenAI, hit$100 million of ARR quite recently. And then Replit, which went from$10 to$100 million of ARR in just the last six months. Yeah, after being at it for a solid 10 plus years. So I also had Amjot on the podcast, I promise. I'll stop now. But that was another great episode. He's such an incredibly thoughtful guy.

9:53And yeah, I mean, the story of Replit was that he was, from what I understand as an outsider and not an investor, pretty flat for many, many years with a little bit of a kind of like hobbyist student kind of user base. And when they launched their agent a few months ago, it's been, well, what you see on the screen here, this dramatic acceleration, such a wonderful story. Yeah, it's fascinating. You know, historically, they were known for being very much in the hobbyist indie developer educational setting. And what's interesting is actually that is the setting where people are now growing up in an AI native way from an engineering standpoint.

10:34And so in many ways, like doubling down and betting on that distribution channel over the long run has actually probably created the moment that they're having. Well said. So anyways, you know, point is just to say, like, this is a very special group of companies that is ramping quickly and has no sign of stopping. And there really is no plateau in sight for innovation. I mean, it's just, you know, for all of your listeners who spend their time on Twitter and TechCrunch and VentureBeat every day can feel quite overwhelming. Yeah, which is something that we as VCs certainly do not do. Never. My argument in this presentation really is that behind all of the marketing gloss, there really is more substance than there ever has been before.

11:12And if you, I hate to say this, but if you go offline and you aren't keeping up with the updates for a week at a time, you really might miss something spectacular and industry shifting. And so far, you know, we've seen really remarkable, tangible results across our network of engineering organizations. So hopefully this can just put some numbers to, at an organizational level, how code generation is really changing engineering process. We've seen a 30 % to 50 % faster throughput. We've seen 12 % increase in PR merges. This is a really important compounding stat here on the bottom left, which is a 17 % increase in the amount of time folks are spending on roadmap versus maintenance and keeping the lights on and support.

11:53And then in my mind, most impressively, 82 % of people we've spoken to are already using AI to write code. And so that is just, again, an adoption curve that is pretty much unprecedented. And I think, you know, we can talk more about why that is. But I think predominantly it lends itself, again, to the audience and the arena of distribution, IDEs, where people live today. These aren't net new interfaces for the most part, very much embedded in people's style of working. And again, with very similar distribution mechanisms and keystrokes and delivery models. And so it's just been a fascinating couple of years.

12:27On that note of people using AI, I guess there's another topic we could cover. But, you know, we've got to pick, we could talk about this for hours. So we've got to pick what we discussed. Yes, but then there's that whole discussion of autonomous agents versus more of a sort of coding co-pilot. And I think it's more shades of gray in between now. It used to be a little bit of a starker kind of distinction not that long ago. Now it looks like the co-pilots are starting to be agenic in many ways. But still, there's like this really interesting question in the industry where for the, you know, the cursors and the lovable on one hand, you have companies like Devon trying to build fully autonomous agents that basically go away and come back with a fully baked product.

13:22And, you know, which by all accounts doesn't seem to be working yet. But that seems to be pointing to a future that is pretty mind blowing. Yeah, I would say today what we're seeing is this spectrum of people and use cases that vary by experience and complexity. And for now, many of the agentic solutions that are truly end to end, you know, complete something are very much being pointed at lower level tasks that I'd say are, quote unquote, more mindless, that have less dependency, complexity, you know, that have less of a need for context and knowledge. And that's worked very well. So, yeah, there really is a diverse array of ways that you can apply AI to this problem.

14:05And I think many of the best organizations and engineering leaders I talk to deploy all sorts of things across their stack, depending, again, on the use case and the people. You know, I would say, despite all of these amazing numbers and stats, I want to pose sort of a historical analogy here to draw to this space, basically to say that with production surges, it tends to be the case that actually a lot of problems emerge. and in their wake, markets follow. Or maybe a punchier way to say that would be with every surge in production, there's just a cleanup crew that naturally comes and a new market and industry that follows in its wake.

14:39Yeah, so if you'll allow me, we're going to go back in time as far back as the 15th century and talk about a couple examples where we've seen this happen and how we might be able to draw some analogies to today. Let's do it. So starting with the Gutenberg press, this really was the advent of our ability to print books at scale. And so in the 1440s in Europe, we went from about 40 to 3.6K pages being printed per day. I think over the next 60 years or so, there were 20 million books in circulation in Europe, up from 3 ,000. Big disruption, right? It put monks out of business. The people that spend their entire lives copying by hand.

15:19Yes, they turned to making chartreuse. So anyways, this sounds like a great thing, right? I would hope we all love books. But actually, like many of the content issues that we see today, there were a lot of challenges that emerged, namely misinformation, mass reproduced quality issues, a bunch of informational overload. And so we saw a bunch of new industries emerge in the wake of the press, which were all the things that we think about when we think about books, printing, publishing houses, editors, libraries now for consumers to deal with the abundance of options that they have, almost serving as physical indexes of the many books that they could access.

16:01And then a bunch of regulation that came around around licensing and censorship. And again, all the things that we associate with content today. Another example, almost 500 years later, was the Ford assembly line. So in the early 1900s, Henry Ford invented the first continuously moving assembly line, and we dropped the time per car assembly-wise by almost 90%. And so over the next 20 or so years, the output of Model T's soared to about 10 million in the U.S. And we went from a society that was mostly horse and carriage, railroad and trolley driven to all of a sudden being a car country. So we still have car challenges today.

16:40But if you can imagine then, they felt much more extreme. Things like infrastructure strain, safety issues, the need to build all of the roads and infrastructure to support this new economy. factory workers' rights, environmental issues. And so again, we saw these industries emerge. I'll highlight the ones that feel very much endemic to auto. So quality inspectors, mechanics, dealerships, gas station attendants. And then on the regulation side, all the licensing, our driver's licenses, our license plates, traffic police, and again, all of the build out over the following 50 years or so of the freeways, highways, roads that we all use and drive on today.

17:17And so there are a bunch of other examples of this in history. I sort of arbitrarily chose those two, but railroads, map making, the postal system. I think smartphones is probably the most recent example where all of a sudden we have tons of compute in our hand and the ability to take as many pictures as we'd like in a day. And in its wake, you've seen a ton of industry emerge, mostly social media, influencer marketing, et cetera. And then all of the regulation around privacy and biometrics that still feels very top of mind today and that has been evolving over the last decade or so. And so, again, I would just posit, as production surges, you see all of these problems come in the wake of that production and then new industries come around.

17:59And so we didn't bury the lead here. If you think about code, I would argue we're very much seeing a similar trend where on the canonical DevOps cycle diagram, you're seeing code commit just accelerate so much. and today at least many of the processes that are downstream of that, our CICD pipelines, our testing suites, our build infrastructure, the way that we think about observability and monitoring has very much stayed the same, but it's kind of breaking. And so I'll talk a little bit more about that. Yeah, what happens next, yeah. On yet another episode of the Mad Podcast, just a couple of weeks ago, we had Brendan Humphries, CTO of Canva, who was talking about that exactly, which is okay.

18:47It's great that you can create code, but we had one guy who submitted a PR that was 50 ,000 lines, and they have a peer review culture, and they basically had to make the point that it was not okay to just lob over the fence 50 ,000 lines, and now good luck. Somebody's got to review it. So yeah, more code, now what? To put some numbers to how these pressure points are starting to begin to explode, we're just seeing a lot more time spent debugging. We're seeing a ton more security vulnerabilities, which we alluded to a little bit earlier when we were talking about how these models have actually been built and on what data.

19:27We're seeing a ton of performance issues emerge as a result. And, you know, I know you did your podcast with Guillermo. I actually did an event with Malte Uble, who's the CTO at Vercel. and yeah and that that event is on our guilds you know since this is turning out to be a shilling episode we can shill the First Mark guild so guilds are is the name for our private communities that we have at First Mark where we have a bunch of people on a per job basis from the portfolio but also from outside of the portfolio and this specific event was from our CTO guild and so you ran a sort of intimate fireside chat with the CTO of our cell, just for context.

20:09So this was a great event for our CTO guild. And Malte, among many interesting things, I think one of the most fascinating things that he said was that most of his great engineers who have been with the company for a while as they've dogfed, you know, V0 and they've used things like Cursor and Windsurf in-house is that many of his great engineers are actually becoming predominantly professional code reviewers. And this notion that as AI cogen has just totally increased the rate at which they're producing and outputting code commits. You need people who almost act as, you know, people sitting in a toll booth, letting cars pass or not pass, very much doing the same thing with code, but people in those seats that have great taste and know what bad or good or great looks like.

20:55And so I thought that was sort of the tip of the iceberg as you think about, you have these large engineering organizations that have all been trained somewhat similarly on how they should all think about working together and what sorts of jobs that certain people should be doing and not doing and how teams should work together. And we've seen this really, really fast shift where individual ICs who used to be, you know, green dots everywhere on their GitHub repo are all of a sudden becoming code reviewers. And so, yeah, I thought that was a really interesting analogy and I'll expand more on it, but.

21:27That's fascinating, right? So you used to have front-end engineers and back-end engineers and full-stack people. and yeah, there might be a future where people are none of that and just everybody's a code reviewer, which opens up this whole conversation about how do you become a good engineer in the future in this new age of AI and how people should be trained and all those things. But let's keep going. You know, we did a study at FirstMark across, I think it was a little over 300 engineering organizations. And we asked them just sentiment, basically, on CodeGen and how it's made them feel about certain metrics that they use to measure their engineering output.

22:08Yeah, so let's go through some of this, including for people that may be listening to this in a podcast-only version. So what are some of the metrics and numbers we have here? Yeah, so I would say everything in purple, unsurprisingly, which is positive, is very much linked to speed and efficiency and velocity. And so most of these stats are measuring in some way or another time, time spent to develop something, the number of times in a certain period that we commit, how often we're deploying. And so all of those things have been very much sped up in this age of AI. But what's interesting is many of those things don't take into account what happens afterwards.

22:50And so again, we could make as many commits as we want, but that's very much just the first step in the DevOps process as it exists today. In blue are most of the sort of negatively impacted areas across the survey that we did, which are mostly reliability and process oriented. And so like take mean time to restore, for example, if something goes down, how long does it take to restore it? Well, all of a sudden, if we don't have provenance over the piece of code that's relevant to something that happened, an incident, it's much harder for us to go back and say, oh, actually, there was a bug over there.

23:26And so that's been what's generally interesting. I think this slide really just goes to say, again, everybody's getting sped up. The amount of commits is skyrocketing. But all the stuff that we measure when we think about efficiency and all the downstream things that we want to happen post-code commit seem to be faltering. Yeah, I love it. It re-anchors the reality of, okay, it's cool to have AI, but now what does that mean, right? Yeah, so I can be a little more specific here in terms of what's breaking. Yeah, let's do it. We can start with security. Put simply, we're just seeing more security vulnerabilities than ever before.

24:02And new types of security vulnerability. Yes, I would say new types of security vulnerabilities, especially given, and this is sort of separate from the conversation about coding, but given deepfakes and all the stuff that's happening in the arena of email right now, certainly on that side, and then also in the world of CodeGen, to our earlier conversation, many of these models have been trained on really large bodies of open source code. And many of those pieces of code just share vulnerabilities and bugs that you can't see coming in the same way because, again, you're not operating with the same level of decisiveness and meticulousness.

24:41And so as a result, you're seeing things like CICD pipelines and build systems completely breaking. We're seeing, I mean, folks have had brittle CICD pipelines for a while now. We've seen a lot of companies emerge over the last decade to sort of ameliorate that and change the way that we think about CI and CD. But the flake rate is just much higher. And that's true of testing too. So flakes basically is just your ability to either trust or not trust a pass-fail result. And if you have a flake, it essentially means something passed and then it failed, but the inputs were the same. And so we're just seeing flakiness generally across code skyrocketing.

25:19I would say build has come under a lot of duress. It's just much harder in a cache code, which a lot of build tools try to do when Cogen is relatively stochastic. And so again, like the notion that given the same set of inputs or similar inputs, you're just not going to get the same set of outputs. And then QA and code review are areas that we've talked about. But we've seen QA processes just completely overwhelmed. And on the code review side, we're just seeing a new focus on code review that is pretty unrelenting, that I'd say has gone from a step in a process to very much a job in and of itself.

25:54Yeah, and I'm sure we're going to talk about it. But obviously the question is, to which extent can AI review AI? Unfortunately, for those people that are viewing the podcast that are engineers or practitioners, all of the problems that I'm talking about are very much our opportunities as investors. And founders. And founders. And we see a ton of them across all of these spaces. So I won't go into every single one, but I'll call out a few. On the security side, so there's a lot of companies, older companies, Fortune 1000 companies that use EOL software. So maybe they're on like a very old version of CentOS or, you know, maybe they're using a old package, old version of an open source library.

26:35And the notion that they could just upgrade is actually quite, it's a difficult thing to do. And so we're seeing a handful of companies that have actually pioneered what's called auto patching. And so the ability just to, instead of like forcing folks to make a potentially breaking change, just patch a project and let things stay status quo and smooth. That's a capability that you'd be shocked. There are thousands of engineers that have spent days, weeks, months trying to just keep something afloat because of an old dependency or an old package. Sounds brittle. Yeah. you know, across, I'll say QA, we're seeing a lot of interesting agentic solutions that are reasoning based on essentially semantic requirements.