#169 Guillermo Rauch: How To Use AI to Improve Web Development

0:00Everything on Vercel scales with how much traffic you get. It works if you have one visit a day or it works for the largest websites on the internet. We're not that concerned about the CPU or the database. What makes the world go round is the thing you've added on top of that. The business logic, the data transformation pipelines, the front end and the user experience. So I think what's going to happen is that right now we're in a very fast moving time where lots of models are being updated. But at some point that's going to slow down. And I think we're going to enter the era that we have today with the cloud where, of course, we're all using all kinds of models, but we're going to go and be more concerned with what are we offering to the end user.

0:42We have to give them a great user experience, first of all. AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power. So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud. Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application, development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing.

1:31And of course, nobody does data better than Oracle. So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, Take a free test drive of OCI at oracle.com slash IonAI. That's oracle.com slash IonAI. Hi, I'm Craig Smith, and this is IonAI. In today's conversation, I had the pleasure of speaking with Guillermo Roche, CEO and founder of Vercel and the creator of Next.js. We discussed the seamless integration of generative AI applications on the Veracell platform and how it's powering the next wave of dynamic and personalized web experiences.

2:30From the intricacies of deploying AI-driven applications to the democratization of AI technology, Guillermo shed light on the evolving landscape where developers are increasingly leveraging AI in their workflows. As we navigate the discussion from the technicalities of AI models to the practicalities of their applications in various industries, it's clear that Vercel is at the forefront of this transformative era. My name is Guillermo Rauch. I'm the CEO and founder of Vercel. I also created Next.js, which is one of the most popular front-end frameworks in the world. It powers a lot of the interfaces of generative AI companies like ChatGPT.

3:19Anytime you go to a web browser, whether on mobile or desktop, and you type in a URL, increasingly a lot of those sites are being served by Next.js and Vercel, especially when they're very dynamic and personalized. Yeah, and actually Next.js, I was reading that you were into JavaScript when you were very young, which says a lot about you. There are not that many people that even as adults get excited about JavaScript. Yeah, yeah. It's funny. When I was very young, I started doing all kinds of businesses online. And I got really interested in open source technologies and communities around Linux and other programming languages.

4:13but one of the things that really stood out to me was when I started doing jobs online for other people I joined this platform called script layers which is a freelancing platform I noticed that I could like solve problems with lots of different programming languages but one programming language JavaScript stood out because it was the only programming language that could run in a web browser. And at the time, I didn't think of it as like some like 4D chess and super smart MBA, but it's almost like an unfair advantage, right? We have 7 billion people. Almost everybody has a web browser in their pocket or their laptop computer.

4:55It's the most dominant software platform in the world. And this programming language is the only one that can run in those web browsers on the client side, like meaning as close as possible to the interaction with the user. So at the time I didn't, this wasn't so obvious, but it did seem to me like JavaScript had this special significant bit. And I became a, an expert in that language, which powers basically every front end experience in the world today. It powers everything from chat GPT to amazon.com to when you type on Google and it produces suggestions, right? Rich, interactive, dynamic experiences are always built with JavaScript.

5:40So the way to think about Vercel, our company is that we took that language and we made it massively accessible to every company on the planet. We built frameworks around it so that you don't have to reinvent the wheel every time you start a new project. And we focus a lot on giving businesses a great ROI. meaning you put something online and it will make it faster. We'll make it more engaging, more personalized so that people return. So we do very well with e-commerce websites, but now increasingly everybody seems to be building AI products on our platform. And that's where I kind of like dove in into this world of AI.

6:20And it seems to be like all I think about and talk about. Yeah. Yeah. So what's the relationship between Next.js and Vercel? And then can you describe, I've seen you describe Vercel as a service. Other people have described it as a platform, but can you talk about that? Yeah, the way to think about it is you program with Next.js. You design and conceive your front-end experience at a website and that web application. But then you have to deploy it so that you can give people a URL to access it, right? So you have to ship it. You have to make it go live. Vercel is the easiest, fastest, and most scalable platform to ship those Next.js applications to the world.

7:08And we support other frameworks that are similar to Next.js as well and have been inspired from Next.js. So in a nutshell, we're really simplifying the problem of shipping something to the cloud. You and I, we met at reInvent in Vegas. Because anybody that tries to use the cloud today, it almost seems like you need a PhD. You need to learn all the different layers of AWS and Google Cloud and Azure. But that's not how developers want to ship applications. And now increasingly, they're building those applications with AI, and the AI really knows how to write your code and so on. So the idea of burdening the developer with provisioning infrastructure in 2024 is not only very costly, but it also slows businesses down.

7:57Let's say that you want to ship an AI initiative for your business. You could take the path of I'll build my own infrastructure. I will configure AWS and do all of that. or it can ship on Vercel and you go from months to weeks to days to minutes to make something live. And so Vercel runs on AWS, right? So it's an alternative to all of the AWS tools, SageMaker, Bedrock, all of that stuff. Correct. It's basically an alternative to figure it all out in this massive marketplace of options, right? When you go to Vercel, we offer you templates. So for example, one thing that's been super successful is we give you the interface of ChatGPT.

8:48We give you the front end of it. We built a clone that's heavily inspired by ChatGPT. Obviously, it's not exactly ChatGPT. But folks can now deploy their own ChatGPT in two clicks. So compare that to the experience of like booting up EC2 instances and figuring out virtual private networks. and private relays and NAT and all that in Linux, right? It almost feels like, you know, I've been spending 20 years learning this stuff. I don't want anybody to have to learn all those like deep details in order to ship an AI application. So going back to that ChatGVT experience, we open source this kind of as an idea.

9:34We had an idea about a year ago. We said like, okay, Chachapiti is cool. A lot of people are going to use it, but enterprises are going to want their own. They're going to want their own models. They're going to want their own retrieval to their own data sets. But most importantly, because the first cell is so concerned with the front-end experience, the brand experience, we also hypothesized people are going to want to embed that experience into their websites and applications and make it their own. their own user interface components, their own style, maybe their own sort of representations of the data so that you're not just rendering text.

10:13You could render buttons. You could render galleries. You can render maps. You can render charts. And that bet really paid off because now we have a lot of customers that have launched. The way to think about it is it's almost like a domain-specific chat GPT that you can create. there is a company called fin tool that built the bloomberg terminal of ai you and i were talking about it last time and you were interested because it sounds so cool uh there's another one called fin chat also on versell think of it as like talking to your stocks right so i think that idea of using natural language to interface with applications is here to stay and it's not just going to be chat gpd everybody's going to want this technology and now versell is basically democratizing it for everybody to launch and deploy and customize in a couple of clicks.

11:03Yeah. And one thing I found interesting is Bedrock, you know, gives you a, it's also, you can deploy across different models or use different models, but it doesn't include GPT-4, or ChatGPT. Vercel, when you say build on top of ChatGPT, that's one of the options, but you can build on any number of models, right? So the way to think about Vercel's architecture, the architecture we brought to market is that we're separating the front end, which is the user experience, from the back end. Typically, software was built very monolithically. I talked to a lot of our customers, for example, Chico's moved from a Oracle monolith that they've been testing in for 20 years to a composable architecture where the part of the website that interacts with the customer is separated and connected over the network to the different services that supply it.

12:10Think of it as like when you sit down at a restaurant, you're being served and the atmosphere is all awesome. And then the backend is the kitchen, right? So we let you plug in whatever backend you want into that front end. So when you use our chat GPT template, for example, we let you connect to GPT-4, we let you connect to Llama, we let you connect to Mistral. And customers, I think from a business defensibility perspective, are very interested in this, right? Because instead of coupling all the pieces, you're basically stuck with that choice for a very, very long time. Now they're finding that Vercel is sort of giving them a lot of freedom.

12:54And this is true not just for AI applications, it's also true for e-commerce, where your storefront is separated from the content provider, the order tracking system, the search engine. So we're going into this world that's highly modularized. And something that came up, actually, I met with Rubrit, who you introduced me to yesterday. And we were talking about using different foundation models to power something that I'm working on. And I was asking, Is it possible to blend models to get outputs from different models and consolidate or maybe do some sort of reconciliation if one model has something that, yeah.

13:47Absolutely. In fact, this is already how a lot of the AI services at scale work. It's kind of like the dirty secret of AI is that sometimes you think you're interacting with one model. because the industry right now is so concerned. It's so funny because it's the days of like pre-cloud or the early days of cloud. The industry is very concerned with models and hardware. And that's strange because when you think about modern cloud and modern applications, we don't know what hardware is running it. And the analogy for the model would be like, we're not that concerned about the CPU or the database.

14:27What makes the world go round is the thing you've added on top of that, the business logic, the data transformation pipelines, the front end and the user experience. So I think what's going to happen is that right now we're in a very fast moving time where lots of models are being updated and replaced and so on. But at some point, that's going to slow down. And I think we're going to enter the era that we have today with the cloud where, of course, we're all using all kinds of models, but we're going to go and be more concerned with what are we offering to the end user. We have to give them a great user experience, first of all.

15:11So going back to your specific question about combining models, and I was mentioning, when you use ChatGPT and other AI products, they're already intelligently routing your query through an ensemble of models. Some thing might be best answered by a smaller model. some type of query might go to another model where also the ui changes when that model responds the best example is like dally which generates images yeah so you enter please draw me a photo of a cat it's almost like you're going through a different pipeline right i think that's awesome because at the end of the day it's about the user experience i don't care if it's one model 10 models 100 models i've heard a lot of our customers are also finding okay i'm starting with a very powerful model that can reason extremely well but then once i notice that there's clusters of queries that the users make i can first send them to a cheaper model i just heard from a customer today who's actually doing first summarization with a smaller model that is really good at summarization.

16:29Retrieval and summarization. Then the summary is handed on to the very powerful reasoning model. So the very powerful reasoning model deals with a smaller context window and fewer tokens. So there's an analogy in human beings, right? If every support request for Vercel got routed to the CEO, it'd be very expensive use of our time, right? I could probably answer a lot of technical questions, et cetera. But over time, we try to like route them to this specific agent, right? Human agent that can best answer that question. Yeah. And in many cases, by the way, they would be better than me because they're so trained on answering a kind of support request that I might use the wrong language.

17:18I might forget a reference to the docs and things like that. So that's what we're seeing with this idea of the ensemble of models. I mean, I understand that about routing different parts of a query to different models that are appropriate or relevant for different tests. But I've been looking across models for historical summaries, for example, and I'm trying to get beyond finding one that has the least amount of hallucination. And if I ask four models the same question, I get four different outputs. Maybe three of them agree, and the fourth one has something that doesn't appear in the other three.

18:06I don't know if that's a hallucination, but it seems if there were an overlay that then could rationalize those four into a single answer and take out the outliers just because. Yeah, you're touching on a couple of really interesting points. Number one, and the biggest uphill battle that AI as a field has today is reliability. Reliability on two fronts. One is just pure service reliability. The infrastructure is so immature that right now it's very expensive to run models and the quality of service is not there. So, for example, Vercel gives you a very precise SLO, service level objective, on latency and availability.

18:56And if any of that gets violated, like, it's a huge anomaly, right? It's like a stop the world kind of thing. When I use AI services, I'm noticing all the time there's degradations. So the field is still pretty immature, right? But then there's reliability on the quality of the answer. Maybe you run the same question three times and two are good answers and the other one is not so good. So that's basically giving space for a new field or sort of category of applications and services to emerge that deal with the monitoring problem. How do you monitor an LLM such that it's always giving high-quality answers?

19:45How do you benchmark the things that you change about the model when you fine tune it, when you change the prompt, when you replace the model? How do you gain confidence as an application developer, as a business owner, that when you modify the prompt or release this new version, things got better? And what does better even mean? is a really difficult question. So there's tools already in the marketplace. There's a company, for example, called BrainTrust that the way you think about it is, and I use this analogy a lot of like software 1.0 and software 2.0. Software 1.0 was what we grew up in. Highly deterministic, classic programming languages, Java, C++, PHP.

20:37In that world, 1 plus 1 is always 2. In the software 2.0 world, where we're not writing the code ourselves, we train a model with data and then we infer. One plus one is two most of the time, which is a weird thing to say because we're now building all this, we're building the future of this thing, right? So how do you kind of like wrap your head around and your arms around that instability of the beast? So in software 1.0, We use testing methodologies. We use to control and verify the software. We use unit testing, integration testing, end-to-end testing, all kinds of testing methodologies. In the software 2.0 world, we're now inventing the analogous services of that.

21:31So, for example, Braintrust, the way to think about it is like unit testing for your LLM apps. it allows you to write kind of and again this is our highly non-deterministic tasks but you evaluate that the ai is mostly on track over time and then you instrument it as well with user feedback that's why in every ai app you notice that there is like a thumbs up and thumbs down and and there's some signal that's coming from the user so that goes back into that system and to your question okay so if three models are saying one thing and another model saying another thing you're going to use platforms like this in order to ascertain which is the right model for you and your task and how can you prompt it or fine-tune it so that over time you have no let's call it recalls right uh and folks are not saying wait that's wrong uh or you turn a customer The customer cancels their subscription saying like, ah, this tells me things are wrong all the time.

22:41Have you used Copilot, by the way, the technology that auto-completes your code? I'm sorry, say it again? So Copilot by Microsoft, Copilot by Microsoft. Yeah, of course, yeah. So Copilot has an interesting signal in it, right? Because it suggests what code you're going to complete with, and then you take it or not take it. So that's like an implicit button of thumbs up and thumbs down. So what the people that are working on those products, including Vercel, do is we monitor the acceptance rate of a suggestion. And that's why you're going to find that the best AI products are the ones that are suggested.

23:23The ones that have actually made it and scaled and produced a lot of revenue are suggested because you know that you're sometimes wrong, but the cost of being wrong is not very high because the user just ignores it. So when I recommend to our engineers, what are the best AI products to work on today? They're the ones that put the human in the loop and they act as the, I write your first draft. And you act as the editor and you say yes or no, and maybe you tweak it a little bit and then you ship it. Yeah. I mean, that's a form of reinforcement learning with human feedback, right? Because there's also that feedback is then training or fine tuning the model.

24:20I've talked to a lot of people about reinforcement learning with AI feedback, which to me sounds a lot more efficient, but also can work in real time. I mean, this human feedback, if I have an application and I'm serving customers an output in real time, I can't ask them, I can't give them a choice and ask them which one they think is best. I need to serve them an output. And if I have someone, a human in the loop, obviously it's not going to operate in real time. Have you looked at this AI feedback mechanism? I know that MIT is doing some research on it. Yeah, I think this is a very active area of research because we still have not figured out what is the best way to scale the reinforcement learning phase of training the models.

25:23And to your point, I think one thing that's really important to call out is when you're iterating on an application, your only tool is not just retraining the model. What you were mentioning earlier is that today is very costly and slow. Taking the model, doing more RLHF, that takes a long time. But there's other tools that developers have in their toolbox. They can modify the prompt. They can augment it with other data. They can chain different models together. So I see it as a critical product metric that not only informs model development, but it also informs to the developer, how do you even present the data to the user?

26:08How do you decorate the prompt and embellish the prompt and add more data to the system over time? But you always have to have that metric and then work backwards to how you improve quality over time. And reliability. I mean, some of what you mentioned is referring to RAG, right? Retrieval Augmented Generation. Correct. To me, that sounds like a short-term solution that eventually models will be, again, through AIF or... Yes. I would love to think so. I would love to call it temporary, but right now, at this point in time, it almost seems like the opposite is true. Because one of the biggest enemies of AI clearly is that reliability problem we're talking about, right?

27:04Alicinations, et cetera. And outdated data, not just that, right? So I'll give you a very concrete example. One of the really cool things that you can build with Vercel is, as I mentioned, we give you all these templates to add AI capabilities into your products. One of our solutions is how do you add search to documentation websites, content websites, so that you can ask an AI to solve a problem for you. And this is really cool because you could imagine that the customer is going to go to ChatGPT, but you don't control ChatGPT. So you don't know if it's up to date with your data, your documentation, your content, your blog posts, et cetera.

27:50And you also get loose the feedback loop. Now, like, because that's happening somewhere else, you don't know if people are getting the answers to their questions. And if the answer is a hallucination or not. So we advocate for, here's all the infrastructure. And we make it really, really easy to, like, build yourself out of your product. And one of the things that we found is you almost should completely distrust the knowledge contained in the raw model that comes from this training data set. Because the way you think about the model is that you're making the model memorize the stuff. and in the process of memorize you're asking you to compress forget certain things and from that process you get reasoning so that means that the objective of training the model was reasoning not memory if the objective was memory it would be a database and if the training data set was one terabyte, the model would be one terabyte or maybe less after GZIP and some compression, right?

29:10Which means that really what you're getting is a resending engine. The data of the world is far larger and the production of that data far outpaces the training process. In fact, the average enterprise probably generates the volume of data that was used for training some of these models per day, per week, per month. So if that relationship continues to hold, that you're producing lots and lots and lots of data, and then you want AI systems that can work with it, because customers will always have real-time questions, right? I already lived through this when I first came to the Valley. MapReduce was all the rage.

30:00big data, et cetera, in lots and lots of offline asynchronous processes. In order to get an answer to a question, you had to wait two weeks. And then snowflake happened. And now everything is real time. We ingest data to the minute. And then we ask with SQL. That's the big data solution today. We're going to face similar pressure with AI. training a model is the map reduce job that takes freaking months weeks whatever but then we're gonna have lots and lots of real-time data and questions and i think rag in from that point of view is probably here to stay for a while i think in this space nothing lasts more than like a couple quarters right now but it's a fairly stable solution right now that you know i have a great like end user example of i use perplexity ai a lot because perplexity has real-time answers and i've kind of trained myself like my muscle memory right now is if i'm asking if i'm working on a reasoning problem like um writing something and summarizing it or a spin or rewriting something I go to chat GPT and I have that back and forth with the model.

31:27But if I want real-time knowledge, my muscle memory is now perplexing all the time in the magic there. I mean, there's probably many pieces to it, but of course they're doing real-time retrieval in order to augment generations. They're not retrieving from a vector database, which a lot of reg is doing. They're retrieving through search, right? I think it's probably a combination. I don't know the details, but the way that I would summarize the transition away from Google into this new systems is the transition from keyword reasoning and keyword targeting to, it's almost like spatial reasoning if you think about it in terms of vectors, right?

32:15You're approaching this space of the concept. and the magic of perplexity is that i don't have to know the exact keyword i can talk to it almost like when i talk to a friend i'm like remember when we watched that thing i think it was a couple years ago we're in san francisco and like like you can talk to the AIs in that kind of like, you know, approximated way. And that to me seems like a durable change because that's more normal and more how we think and how we interact with the world. Whereas it almost feels like Google is truly a tool with a learning curve. Like you had to think about it in terms of like, I know how to Google, which is still true today.

33:09I'm a pretty good Googler, but I don't know if that's a durable skill. I probably knew taking off my LinkedIn. Yeah. So at the very beginning, you were talking about writing in Next.js and then deploying on Vercel. But Vercel, you can write in any language. It doesn't require Next.js, does it? It doesn't require Next.js. we have support for over 35 frameworks that are custom tailored to writing that user interface side of thing, right? Our focus is still on, for example, I just met a customer who wants to build the fastest checkout experience on the internet. That's our joint mission. That's why they're partnering with Vercel.

33:56And they're choosing one of these frameworks because that's how you do it today. One of the things that we're trying to disabuse ourselves from as a global community of developers is that every single project requires building your own tools. That's the old way of using the cloud. You sit down, you always start from scratch, from an empty canvas. I've met countless enterprises and developers who in the past were building their own frameworks prior to starting projects. And the way to think about it is, I used to joke, the problem with building your own tools is not just all the sunk cost, is that your developers will not have stack overflow when something goes wrong.