Andrew Ng: Building Faster with AI

0:00It's really great to see all of you. What I want to do today, since this is Build as Startup School, is share with you some lessons I've learned about building startups at AI Fund. AI Fund is a venture studio, and we build an average of about one startup per month. And because we co-founded startups, we're in there, writing code, talking to customers, designing our features, determining pricing. And so we've done a lot of reps of not just watching others build startups, but actually being in the weeds, building startups with entrepreneurs. And what I want to do today is give you some of the lessons I've learned building startups, especially around this changing AI technology and what it enables.

0:38And we focus on the theme of speed. So it turns out that for those of you that want to build a startup, I think a strong predictor for startups' odds of success is execution speed. And I have a lot of respect for the entrepreneurs and executives that can just do things really quickly. and new AI technology is enabling startups to go much faster. So what I hope to do is share with you some of those best practices, which are frankly changing every two to three months still to let you get that speed that hopefully lets you have a higher odds of success. Before diving into speed, you know, a lot of people ask me, hey, Andrew, where are the opportunities for startups?

1:17So this is what I think of as an AI stack, where at the lowest level are the semiconductor companies, then the clouds, our hyperscalers built on top of that. A lot of the AI foundation model companies built on top of that. And even though a lot of the PR excitement and hype has been on these technology layers, it turns out that almost by definition, the biggest opportunities have to be at the application layer because we actually need the applications to generate even more revenue so that they can afford to pay the foundation, cloud, and semiconductor technology layers. So for whatever reason, media and social media tends not to talk about the application layer as much.

1:57But for those of you thinking of building startups, almost by definition, the biggest opportunities have to be there. Although, of course, the opportunity is at all layers of the stack. One of the things that's changed a lot over the last year, in terms of AI tech trends, if you ask me what's the most important tech trend in AI, I would say is the rise of agentic AI. and about a year and a half ago when I started to go around and give talks to try to convince people that AI agents might be a thing, I did not realize that around last summer, a bunch of marketers would get a hold of this term and use it as a sticker and slap it on everything in sight, which made it almost lose some of its meaning.

2:36But I want to share with you from a technical perspective why I think agentic AI is exciting and important and also opens up a lot more startup opportunities. So it turns out that the way a lot of us use LLMs is to prompt it, to have it during an output. And the way we have an LLM output something is as if you're going to a human, or in this case an AI, and asking it to please type on an essay for you by writing from the first word to the last word all in one go without ever using backspace. And humans, we don't do our best writing being forced to type in this linear order. And it turns out neither does AI, But despite the difficulty of being forced to write in this linear way, our elements do surprisingly well.

3:18With agentic workflows, we can go to AI system and ask it to please first write an essay outline, then do some web research if it needs to, and some web pages to put in the own context, then write the first draft, then read the first draft and critique it and revise it and so on. And so we end up with this iterative workflow where your model does some thinking and some research, does some revision, goes back to do more thinking. And by going around this loop many times, it is slower, but it delivers a much better work product. So for a lot of the projects that AI fund has worked on, everything from pulling out complex compliance documents to medical diagnosis, to reasoning about complex legal documents, we found that these agentic workflows are really a huge difference between working versus not working.

4:04But a lot of the work that needs to be done, a lot of the valuable businesses to be built still, will be taking workflows, existing or new workflows, and figuring out how to implement them into these types of agentic workflows. So just to update the picture for the AI stack, what has emerged over the last year is a new agentic orchestration layer that helps application builders orchestrate or coordinate a lot of calls to the technology layers underneath. And the good news is the orchestration layer has made it even easier to build applications. But I think the basic conclusion that the application layer has to be the most valuable layer of the stack still holds true.

4:43With a bias or focus on the application layer, let me now dive into some of the best practices I've learned for how startups can move faster. It turns out that at AI Fund, we only focus on working on concrete ideas. So to me, a concrete idea, a concrete product idea is one that specified in enough detail that an engineer can go and build it. So for example, if you say, let's use AI to optimize healthcare assets, you know, that's actually not a concrete idea. It's too vague. If you tell me to write software to use AI to optimize healthcare assets, different engineers would do totally different things.

5:21And because it's not concrete, you can't build it quickly and you don't have speed. In contrast, if you had a concrete idea, like let's write software to let the software hospitals let patients book MRI machine slots online to optimize usage. I don't know if this is a good or a bad concrete idea. It's actually business already doing this. But it is concrete and that means engineers can build it quickly. If it's a good idea, you find out that it's not a good idea, you will find out. But having concrete ideas buys you speed. Or someone would say, let's use AI for email personal productivity. Too many interpretations of that.

5:51That's not concrete. But if someone says, can you build an app? Gmail integrate the automation that uses the right prompts, or it's my filter and tag emails. That is concrete. I could go build that this afternoon. So concreteness buys you speed. And the deceptive thing for a lot of entrepreneurs is the vague ideas tend to get a lot of kudos. If you go and tell your friends, we should use AI to optimize the use of healthcare assets, everyone will say that's a great idea. But it's actually not a great idea, at least in the sense of being something you can build. It turns out when you're vague, you're almost always right.

6:23But when you're concrete, you may be right or wrong. Either way, it's fine. we can discover that much more fast, which is what's important for a startup. In terms of executing concrete ideas, I find that at AI Fund, I ask my team to focus on concrete ideas because a concrete idea gives clear direction and the team can run really fast to build it and either validate it, prove it out, or falsify it and conclude it doesn't work. Either way is fine, so let's do that quickly. And it turns out that finding good concrete ideas usually requires someone, it could be you, it could be a subject matter expert, thinking about a problem for a long time.

6:59So for example, actually before starting Coursera, I spent years thinking about online education, talking to users, holding my own intuitions about what would make a good edtech platform. And then after that long process, I think YC sometimes calls it wandering the idea maze. But after thinking about it for a long time, you find that the guts of people that have thought about this for a long time, can be very good about rapidly making decisions. As in, after you've thought about this, talk to customers and so on for a long time, if you're asked as experts, should I build this feature or that feature, you know, their gut, which is an instantaneous decision, can be actually a surprisingly good proxy.

7:36It can be a surprisingly good mechanism for making decisions. And I know I work on AI. You might think I'll say, oh, we need data. And of course, I love data. But it turns out getting data for a lot of startups is a slow mechanism for making decisions and a subject matter expert with a good gut is often a much better mechanism for making a speedy decision. And then one other thing, but many successful startups at any moment in time, you're pursuing one very clear hypothesis that you're building out and trying to sell, trying to validate a false fight. And a startup doesn't have resources to hedge and try 10 things at the same time.

8:11So pick one, go for it. And if data tells you to lose faith in that idea, that's actually totally fine. Just pivot on a dime to pursue a totally different concrete idea. So that's what often feels like an AI fund. We're pursuing one thing doggedly with determination until the world tells us we were wrong, then change and pursue a totally different thing with equal determination and equal doggedness. And one of the patterns I've seen, if every piece of new data causes you to pivot, it probably means you're starting off from too weak a base of knowledge, right? If every time you talk to a customer, you totally change your mind, probably means you don't know enough about that sector yet to have a really high-quality concrete idea.

8:50And finding someone that's thought about a subject for longer may get you on the better path. In order to go faster, the other thing I often think about is the build feedback loop, which is rapidly changing when it comes to how we build with AI coding assistance. So when you're building a lot of applications, one of the biggest risks is customer acceptance. A lot of startups struggle, not because we can't build whatever we want to build, but because we build something and it turns out, nobody cares. And so for a lot of the way I build startups, especially applications, less so deep tech, less so technology startups, but definitely application startups, is we often build software.

9:26So this is an engineering task. And then we will get feedback from users. And this is a product management task. And then we'll go back. Then basically user feedback, we'll tweak our views on what to build, go back to write more software. And we go around this loop many, many times, iterate toward product market fit. And it turns out that With AI coding assistance, which André talked about as well, rapid engineering is becoming possible in a way that just was not possible. It's becoming much more feasible. So the speed of engineering is going up rapidly, and the cost of engineering is also going down rapidly.

10:02This changes the mechanisms by which we drive startups around this loop. When I think about the software that I do, I maybe put it into two major buckets. Sometimes I build quick and dirty prototypes to test an idea. You can say, build a new customer service chatbot. Let's build AI to process legal documents, whatever. Build a quick and dirty prototype to see if we think it works. The other type of software where I do is, right, maintain production software, maintain legacy software, with these massive production-ready code bases. Depending on which analysts report you trust, it's been hard to find very rigorous data on this.

10:33You know, when writing production quality code, maybe we're 30 to 50 % faster with AI systems. Hard to find a rigorous number, maybe. It's possible to be. But in terms of building quick and dirty prototypes, we're not 50 % faster. I think we're easily 10 times faster, maybe much more than 10 times faster. And there are a few reasons for this. When you're building standalone prototypes, there's less integration with legacy software infrastructure, legacy data needed. Also, the requirements for reliability, even scalability, even security are much lower. And I know I'm not supposed to tell people the right insecure code, right?

11:09It feels like the wrong thing to say, but I routinely go to my teams and say, go ahead, write insecure code. Because if this software is only going to run on your laptop, and you don't plan to maliciously hack your own laptop, it's fine to have insecure code. But of course, after it seems to be working, please do make it secure before you ship it to someone else. And, you know, like a leaking PII, a leaking sensor data, that is very damaging. So before you ship it, make it secure and scalable. But if you're just testing it, it's fine. And so I find increasingly, startups will systematically pursue innovations by building 20 prototypes to see what works.

11:43Because I know that there's some angst in AI, a lot of proof of concepts don't make it into production. But I think by driving the cost of a proof of concept low enough, it's actually fine if lots of proof of concepts don't see the light of day. And I know that the mantra, move fast and break things, got a bad rep. Because, you know, it broke things. And some teams took away from this that you should not move fast, but I think that's a mistake. I tend to tell my teams to move fast and be responsible, and I think they actually lost the ways to move really quickly while still being responsible. And in terms of the AI assistance coding landscape, I think was it three, four years ago, Code Autocomplete, popularized by GitHub Copilot.

12:25And then there was a cursor Windsurf generation of AI-enabled IDEs. We've been great to use Windsurf and cursor quite a lot. And then starting, I don't know, six, seven months ago, there started to be this new generation of highly agentic coding assistants, including using O3 a lot for coding. Cloud Code is fantastic. Since Cloud 4 released, it's become... And honestly, again, in three months, I may use something different. But the tools are evolving really rapidly. But I think Cloud Code Codex is a new generation of highly agentic coding assistants that is making developer productivity keep on growing.

13:01And the interesting thing is if you're even half a generation or one generation behind, it actually makes a big difference compared to if you're on top of the latest tools. And I find my team is taking really different approaches to software engineering now compared to even three or six months ago. One surprising thing is we're used to thinking of code as this really valuable artifact because it's so hard to create. But because the cost of software engineering is going down, code is much less of a valuable artifact as it used to. So I'm on teams where we've completely rebuilt a code base three times in the last month.

13:30Because it's not that hard anymore to just completely rebuild a code base, pick a new data schemer, it's fine. Because the cost of doing that has plummeted. Some of you may have heard of Jeff Bezos' terminology of a two-way door versus a one-way door. A two-way door is a decision they can make. If you change your mind, come back out, reverse it relatively cheaply. Whereas a one-way door is you make a decision and you change your mind. It's very costly and very difficult to reverse. So choosing the software architecture of your tech stack used to be a one-way door. Once you're built on top of a certain tech stack, you set a database schema, really hard to change it.

14:03So that used to be a one-way door. I don't want to say it's totally a two-way door, but I find that my team will more often build on a certain tech stack. A week later, change your mind, let's throw the code base away and redo it from scratch on the new tech stack. I don't want to over-hype it. We don't do that all the time. There are still costs to redoing that. But I find my team is often rethinking what is a one-way door and what's now a two-way door because the cost of software engineering is so much lower now. And maybe going a little bit beyond software engineering, I feel like it's actually a good time to empower everyone to build AI.

14:36Over the last year, a bunch of people have advised others not to learn to code on the grounds of AI were automated. I think we'll look back on this as some of the worst career advice ever given. Because as better tools make software engineering easier, more people should do it, not fewer. So when many decades ago, the world moved from punch cards to keyboard and terminal, that made coding easier. When we moved from assemblies, high level languages like COBOL, there were actually people arguing back then that now we have COBOL, we don't need programmers anymore. People actually wrote papers to that effect.

15:11But of course that was wrong and programming languages made it easier to code and more people learned to code. Texts are just the IDEs, the AI coding assistant, and as coding becomes easier, more people should learn to code. I have a controversial opinion, which is I think it's time for everyone in every job role to learn to code. And in fact, on my team, my CFO, my head of talent, my recruiters, my front desk person, all of them know how to code. And I actually see all of them performing better at all of their job functions because they can code. And I think I'm probably a little bit ahead of the curve.

15:46I'm probably most businesses are not there yet. But in the future, I think we'll empower everyone to code. A lot of people can be more productive. I want to share with you one lesson I learned as well on why we should have people learn to do this, which is when I was teaching a Genesive AI for everyone on Coursera, we needed to generate background art like this using Mid Journey. And one of my team members knew art history. And so he could prompt Mid Journey with the genre, the palette, the artistic inspiration, had a very good control over the images he generated. So we ended up using all of Tommy's generated images.

16:22Whereas in contrast, I don't know art history. And so when I prompt image generation, I could write please make pretty pictures of robots for me. And I could never have the control that my collaborator good. And so I couldn't generate as good images as he could. And I think with computers, one of the most important skills of the future is the ability to tell a computer exactly what you want so they will do it for you. And it will be people that have that deeper understanding of computers that will be able to command the computer to get the outcome you want. And learning to code, not that you need to write the code yourself, steer AI to code for you, it seems like it would remain the best way to do that for a long time.

17:02With software engineering becoming much faster, the other interesting dynamic I'm seeing is that the product management work, getting user feedback, deciding what features to build, that is increasingly the bottleneck. And so I'm seeing very interesting dynamics in multiple teams. Over the last year, a lot more of my teams have started to complain that they're bottlenecked on product engineering and design because the engineers have gotten so much faster. Some interesting trends I'm seeing, Three, four, five years ago, Silicon Valley used to have these slightly suspicious rules of thumb, but nonetheless rules of thumb.

17:34We'll have 1 p.m. to four engineers or 1 p.m. to seven engineers. It was this like p.m. product manager to engineering ratio, right? Which would take a grain of salt, but it was typical of 1 p.m. to six, seven engineers. And with engineers becoming much faster, I don't see product management work, designing what to build, becoming faster at the same speed that engineering is. I'm seeing this ratio shift. So literally yesterday, one of my teams came to me and for the first time, when we're planning a head comfort project, this team proposed to me not to have 1 PM to four engineers, but to have 1 PM to 0.5 engineers.

18:09So the team actually proposed to me. I still don't know if this is a good idea. For the first time in my life that I saw managers propose to me having twice as many PMs as engineers. It was a very interesting dynamic. I still don't know if this proposal I heard yesterday is a good idea, but I think it's a sign of where the world is going. and I find as a PMC in code, or engineers with some product instincts often end up doing better. The other thing that I found important for startup leaders is because engineering is going so fast, if you have good tactics for getting rapid feedback to shape your perspective on what to build faster, that helps you get faster as well.

18:44So I'm going to go through a portfolio of tactics for getting product feedback to keep shaping what you will decide to build. And we're going to go through a list of the faster, the maybe less accurate, the slower, more accurate tactics. So the fastest tactic for getting feedback is look at the product yourself and just go by your gut. And if you're a subject matter expert, this is actually surprisingly good, if you know what you're doing. A little bit slower is go ask three friends or teammates to get feedback, to play your product and get feedback. A little bit slower is ask three to ten strangers for feedback.

19:18It turns out when I build products, one of the most important skills I think I learned was how to sit in the coffee shop, how to sit in there. When I travel, I often sit in the hotel lobby. It turns out I learned to spot places of high foot traffic and very respectfully grab strangers and ask them for feedback on whatever I'm building. This used to be easier when it's less known. When people recognize you, it's a little bit more awkward. But I found that I've actually sat with teams at the hotel lobby, very high foot traffic, and very respectfully asked strangers, Hey, we're building this thing.

19:51Do you mind taking a look? Oh, and I actually learned in the coffee shop that a lot of people are working. A lot of people don't want to be working, so we give them an excuse to be distracted. They're very happy to do that too. But I've actually kind of made tons of product decisions in the hotel lobby or the coffee shop with collaborators, just like that. Send prototypes to 100 testers. if you have access to logic, there's a user, send prototypes to more users. And these get to be slow and slower tactics. And I know Silicon Valley, we like to talk about A-B testing. Of course, I do a ton of A-B testing.

20:22But contrary to what many people think, A-B testing is now one of the slowest tactics in my menu because it's just slow to ship it, depending on how many users you have. And then the other thing is, as you use anything but the first tactic, some teams will look at the data and make a decision. But the missing piece is when I A-B test something, I don't just use the result of A-B test to pick product A or product B. My team will often sit down and look carefully at the data to hone our instincts, to speed up, to improve the rate at which we're able to use the first tactic to make high-quality decisions.

20:58I often sit down and think, gee, I thought this product name will work better than their product name. Clearly, my mental model they use is wrong. So really sit down and think to update our mental model using all of that data to improve the quality of our guts on how to make product decisions faster. That turns out to be really important. All right. So talked about concrete ideas, speed up engineering, speed up product feedback. This is one last thing I want to touch on, which is I've seen that understanding AI actually makes you go faster. And here's why. As an AI person, maybe I'm biased to be pro-AI, but I want to share you why.

21:33So it turns out that when it comes to mature technology like mobile, you know, many people have had smartphones for a long time. We kind of know what a mobile app can do, right? So many people, including non-technical people, have good instincts about what a mobile app can do. If you look at mature job roles, like sales, marketing, HR, legal, they're all really important, all really difficult. But, you know, there are enough marketers that have done marketing for long enough and the marketing tactics haven't changed that much in the last year. So there are a lot of people that are really good at marketing.

22:02And it's really important, really hard, but that knowledge is relatively diffused because, you know, the knowledge of how to do HR, it hasn't changed dramatically, you know, in the last six months. But AI is emerging technology. And so the knowledge of how to do AI really well is not widespread. And so teams that actually get it, that understand AI, do have an advantage over teams that don't. Whereas if you have an HR problem, you can find someone, you know, that knows how to do it well, probably. But if an AI problem, knowing how to actually do that could put you ahead of other companies. So things like what accuracy can you get for a customer service chatbot?

22:37You know, shoot your problem or fine tune using a genetic workflow. How do you get a voice out to low latency? There are a lot of these decisions that if you make the right technical decision, you can solve the problem in a couple of days. If they make the wrong technical decision, you could chase a blind alley for three months. And one thing I've been surprised by, it turns out if you have two possible architecture decisions, is one bit of information. It feels like if you don't know the right answer, at most you're twice as slow, right? One bit, you know, try both. It feels like one bit of information can at most buy you a two-way speed up.

23:09And I think in some theoretical sense, that is true. But what I see in practice, if you flip the wrong bit, you're not twice as slow. You spend like 10 times longer chasing a blind alley, which is why I think going in to have this right technical judgment, it really makes startups go so much faster. The other reason why I find staying on top of AI really hopeful for startups is over the last two years, we have just had a ton of wonderful Gen AI tools or Gen AI building blocks, right? Partial lists, but prompting, agentic workflows, evals, guardrails, RAG, voice app, async programming, lots of ETL, embeddings, fine tuning, graph DB, computer use, MCB, VZ models.