Embedding AI: The Questions Every CEO is Asking

0:00It is not true that language models make search engines unnecessary if anything they make the search engines more valuable because now all that data that you can search becomes like 10x more powerful. 2022 was a break out year for AI as several new tools gained mass adoption. In fact, many have even claimed that ChatGBT is the fastest growing app of all time. And despite machine learning being embedded in many applications already, for millions of users, these tools felt like their first real -world encounter with AI. And that's because tools like ChatGbC or Mid -Journey have put AI at the forefront.

0:42But looking ahead, that may not be the case. The same way that your users don't care if your web app was built with Angular or React or if it happens to be running on AWS or Heroku, the use of AI alone will not be enough to win over users in the long run. Instead, there will be a whole host of ways that companies differentiate as they cleverly embed AI with a nod toward solving their customers core problems. And that's precisely what seems to be the topic of conversation in every boardroom. CEOs are asking how to best integrate this new superpower. But they're also asking important questions around data privacy, competition, cost, accuracy, and also doing all of this really quickly.

1:28How do you structure the data? How do you train them up? How do we make them more efficient, more insightful, more impactful? And also why haven't we shipped this yet? And in today's episode, we speak with three different companies tackling exactly this, exploring the unique challenges and considerations of implementing AI into their existing products, and what they've learned so far from this fast -moving platform shift. As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.

2:06Please note that A16Z and its affiliates may also maintain investments in the company's disgust in this podcast. For more details, including a link to our investments, please see a16c .com slash disclosures.

2:30Now, before we get into the weeds of how each company is implementing AI, we wanted to start our conversation with Z, Inam, co -founder and CEO of CRESTA. Cresca is bringing real -time intelligence in AI to contact center. And here, Zed reminds us of the upside of AI and what it can truly unlock. So I read that contact centers have 30 to 45 % attrition in the first year with agents, and in some cases that can even be as high as 80%. So what is going on in these contact centers? Why is the Trishon so high? that employee network from Motorsport for contact centers often less than zero. So generally, it's not a role that folks sort of end up seeking with for a long time.

3:13For a few different factors, sort of seasonal demand for increased contact center volume, you have relatively low wages and the sort of high inflation environment. And then you have overall sort of pretty high stress job. It's like sort of taking phone calls, frustrated customers and customers in that kind of environment. Those things add up and lead to a relatively higher turnover environment. Absolutely. I feel like most people don't need a reminder of how brutal that job can get because they've been on the other side. Now many people jumped at thinking that AI should replace the contact center in customer service agents.

3:47But Cresta instead has its eye on transforming a historically low -NPS job into one of mastery and creativity. When you look at artificial intelligence, I think there's really two ways to look at it. There's one way to look at it, which is lazy artificial intelligence. And that's basically, I have an existing process that my business does right now. I'm going to take that process end to end and I'm going to automate it. And AI is really good at that. There's another way to leverage AI, which is more like an creative approach. It was just understanding my job or my role as a business is to deliver the best possible customer experience or the best possible product experience.

4:22And so like the information and the knowledge that's in my conversations with my customers as one, just really great ways to build strong relationships with my customers. But two is just a gold mine of data information about my product, my market, my competitors, what's happening, how the world is changing. Can you give a couple examples of that? What can we do with this technology that a human alone could not effectuate? Yeah, so a great example of it is when you have thousands and hundreds of thousands of conversations with your customers, then nuggets of information about product feedback that you can use to sort of improve your product or improve your sort of how you position your products in the market or how competition is changing.

5:01Specifically, like we work with companies, our large telecommunications companies that sort of are constantly looking at evaluating how are the pricing and packaging. They're very small and bundles or very cable bundles and bundles. And there's just a lot of nuance and data and understanding, okay, how does my customer perceive, but I make this offer? for what is the context that they're coming in with and that really informs their market strategy in terms of what's the new product that they should launch, what's the new package or pricing that they should bring to market, like that's a way that a company can accelerate its development based on just like having a very, very close like year to the ground and like a real strong policy like the customer is same.

5:36And that's ultimately the best way to build companies and products. And artificial intelligence is amazing at that because like what's possible now with large language models, what's possible now with like sort of very advanced deep learning is that you can summarize, you can sort of synthesize, you can pull together information and context in ways that they can take huge amounts of data, make it super simple to understand and get to insight and get to understanding really quickly, in ways that the conversations were unstructured, not possible. Like, they were sort of on legacy on premise, like audio files that you'd have to listen to one by one, this different 20 hours to figure out what that's going on.

6:09And now you have the super advanced human level speed transcription to human level summarization, human level question answering that helps these companies get to the next level of iteration as a business. What you're saying is actually really fascinating. So when I thought of customer service, I thought, okay, this AI will actually be able to help converse with the customer. But what it sounds like you're saying is it also provides the business with an additional layer of understanding because it can basically go through all of this unstructured data and gain insights from it. like you said, if some cohort of customers is on one plan, and they notice something about the conversations that they can now parse through, again, this large bucket of unstructured data and go and basically back to the company and say, hey, maybe you should start a plan of this nature, maybe you should offer this cohort of customers, this deal, maybe you should talk to them in this way.

6:57Is that basically what you're saying? Yeah, and that's like the missing piece of all this is that it's a bidirectional thing, right? It's like you're serving the customer, but you're also serving the business, and it would be part of information in all this. For me, the most inspiring example of this is actually Andy Grove. Intel was originally a memory business for a very long time and it started as a memory business. And then there was like what Andy calls a strategic inflection point where there was a 10x difference in the cost of memory production. And so all these Japanese manufacturers were sort of able to produce memory in like this 10x difference.

7:27And it sort of completely changed the dynamics of that market. And it's funny like the way he phrased it. But the salespeople at Intel came back to headquarters Japanese salespeople and they would say the customers are no longer as respectful as they were. Like that was like the first signal that he got that something is different to the market, but like that was an input to him that ultimately led to him dramatically pivoting and changing the strategy of the company to become a microprocessor company. Because they realize that the market is completely changing, customers are changing, customer reception to their products is changing into their salespeople and that they need to hard pivot the company and to become a microprocessor company.

8:01It's a hard decision to make, but when you have the data, like when you understand what's going on with your customers, you then have the data you're informed to make the right product companies for any decisions, that ultimately save the company. So it turns out that these AIs won't just support your customers, but they can actually be an ear or a thousand ears to the ground, feeding data back to your company, informing how you can better serve them. Now we'll come back to the contact center of the future, but now let's introduce Barry McCartle, co -founder and CEO of Hex. Hex is like a sculptors tool for data, molding, shaping, refining data, devaluable insights and visualizations in various formats, or in various words.

8:40The platform for collaborative data science and analytics. Let's see a little bit more about what the Hex team is building into their product. Earlier this year we launched the closed beta of what we call Hex Magic. Magic is basically a set of AI tools built right into the Hex UI. And so it lets you generate an edit code. For example, this morning I asked it, what's our account of paying customers broken down by pricing tier with some of revenue? And it wrote me a SQL query that did that. Or you can say, refactor this Python code to be a function. And it will do that for you. It also has features to document your codes, which is really useful if you're staying at a super complex query or something that someone else wrote.

9:15It's like, what's going on with this? You can ask it. I'll tell you. Sounds pretty magical if you ask me. But it's one thing to dream up these new AI features and another to successfully roll them out. And as companies like HECS, Cresca, and Sunil here from Source Graph do exactly that, they're inevitably running into challenges. One issue we've probably all experienced at some point is when technology just does not have the right contextual understanding. Like being caught in a customer chat -out loop that just is not addressing your query, something that Cresca knows all too well. We've all been there when we're interfacing with a virtual agent, and it's asking you the same question over and over and you're like, that is not my problem.

9:58And clearly my problem has not been coded into the potential set of solutions. And I find myself in those cases just writing agent agent agent because I need someone with more of that broad context or the ability to interface in a more dynamic way. But I could also see how that could introduce maybe some unexpected responses. I mean, I actually saw someone on LinkedIn today post that he was talking to a virtual service agent and they called him beautiful soul and he's like who would do that? What a human would do that? Soon we'll hear from Barry and also our third guest be young about how we can actually improve the outputs of these models.

10:35But first here's said commenting on how the utilization of these models if implemented incorrectly could really impact your brand. What are you seeing in terms of in the field how different companies are gut checking what's coming out of these LLMs and interfacing with their customers. And I'll also tack on a question, what are people doing in terms of ascertaining whether they should actually disclose whether it's a bot or not? Because as we are getting closer to that idea of human level parity, should we be telling people it's still a bot? Or is it actually good enough where we don't have to anymore?

11:13I think at the end of the day, if you are a brand, you are fundamentally building trust with your subscriber or customer base, right? And your brand value is like sort of, hey, I can like call up this company and like they'll take care of me or like I can trust them. And so like in that case, like I do think leaning towards transparency is the best sort of thing there because long term you want to sort of be known as a company that like doesn't mess around or like you're direct, you tell the customers as it is. And it becomes like an interesting topic as well on the topic of accent masking, right?

11:42Because like that's another use case of the technology that sort of coming to market where folks like uses technology to sort of mass accents or change accents or these kinds of things. And it's the same thing, right? Where like if you're not upfront about it and like there's like some edge case that comes up and it becomes like clear, do you want to take that risk to your brand reputation or not? So in addition to the right disclosures, we can actually improve the performance of these models relative to competitors even if we're using the same underlying models. How do we do that? Through fetching the right data.

12:13Because it turns out that if AI doesn't have access to the right data, it'll just make something up. Garbage in, garbage out, value in, value out. Now here is Beyond View. CoFounder and CTO Source Graph, a code search and navigation tool for deaf teams. This is kind of like general purpose, forced code understanding engine. Here is how Source Graph is thinking about the importance of locating the right information, a problem that they've been working on since inception. So our first kind of major push, I would say, is this editor extension called Cody. And essentially what it does is it's a chap -based interface, but also allows you to search for stuff and context in the code.

12:55And the idea is that we wanted something in our editors that took full advantage of the power of language models, but also addressed a lot of the challenge that people have encountered with large language models, namely the tendency to hallucinate facts when they don't really know the answer. And so that's a place where we thought we could be uniquely positioned to help because Source Graph, you know, with all the piece of context that we have around searching for code and finding references and verifying things actually exist. We are kind of like the perfect fact checker, if you will, for the language model and perfect like relevant context provider to the language model.

13:31And I mean, let's double click on that because there are other tools that help you build code using these language models. You know, just throw out a couple. A lot of people are familiar with GitHub. Copilot, a lot of people are familiar with what Replet is doing with Ghost Rider. But maybe you could actually speak to this idea of fetching the right information. Like, how would something like a copilot do that? And how would something like a Cody actually differentiate? I think the way we think about it, as far as I know, Cody is the only AI -nibbled editor assistant or coding tool today that fetches contexts broadly as we do.

14:04You can ask Cody a question like, hey, you know, Where is the SAML off provider to find my code base? Or where's the GraphQL search API defined? And Cody will actually go and convert that to a couple of search queries, the source graph. And surface these relevant snippets of both code and documentation use that as concrete references to answer the user's question. And that's in contrast to the way that co -pilot works today. It's purely autocomplete driven. And the context that they fetch to do that autocompletion is recent files that you've opened in your editor. So it's kind of this very local context, which works amazingly well.

14:41I mean, huge credit to that team. We think that the next evolution of that is providing more relevant context. And essentially emulating what a human kind of does when you're trying to write code. You as a human, you might go back through some recent history in your editor to see, like, OK, how does that code work? How do that code work? And use that as a pattern matching reference point for the thing that you're currently writing. But more often than not, I think you're doing stuff like go to definition, find references. Let me see a couple of examples of how to use this particular API that I just imported.

15:09I think that that's going to lead to much better results. I think it's also going to lead to much more introspectible results. So getting beyond this, like, oh, elements are magic. How do they work? Is it AGI? You know what? Cody will actually tell you, like, hey, I read these files. And these are the files I'm using to Jerry and answer. And if it completely returns a lie or is wrong, you can usually tell by looking at the context that it read. Like, why are you reading that flat, Cody, that's dumb? And you can like thumbs down that, and it will take that as a reference point to improve the product later.

15:39Another key question that founders are asking is how they'll differentiate. And source graph thinks that open source is part of that answer. The other major distinction I would draw from a lot of other offerings on the market is we are trying to build this as much as the open as possible. So like Cody, the other extension, we've released as open source under the patchy two license. And we think that's kind of like the right mentality to build the standard AI editor assistant that every dev should use. We think that there's like a natural inclination among developers. I'm a developer myself. I prefer to use open tools.

16:11And I think it will make it possible for us to build like a much more like pluggable ecosystem. Like when you think about like the wide wide world of like contexts that you might want to pull in to answer a question in your mind as a developer, it's not just the source code or the markdown documentation. Like maybe you're searching through your issue tracker. Maybe you're searching through like chat messages. Maybe you're searching through like Google Docs and Ocean for like the latest products back. By virtue of being open source, we make it much better and friendly kind of ecosystem into which we can plug in like other developer tools and other pieces of context that aren't necessarily tied to a proprietary compute platform.

16:49Yeah, I mean something that's jumping out to me is if people aren't developers, like even think about writing an article, right? Imagine writing that article based on just like your last five tabs that you had open. That's going to be very different to being able to actually channel your goal and search the web yourself, search your own notes that are specifically likely relevant to the task at hand. And so I think this is like a really interesting perspective as to we know what goes into the model matters in terms of what you get out of the model. And basically what you're establishing is two different approaches, both the open source approach, which allows more people to help you extend it.

17:28And then also this approach of really finding the most relevant information that's going to help the model give you the best answer. So we learn from B Young that what you feed models really matters. And that may be an opening for differentiation relative to competitors. But part of that equation is just the data that you have access to. Here's that commenting on the value of creating proprietary data sets. The opportunity that exists in the market is that you can do that exact same thing. If you can collect proprietary data sets that are unique and at that same scale. And you can actually sort of train foundation models that are able to do even more.

18:05So like the internet or the web pages are like fairly static sort of language modeling task or you're sort of doing a task and you're trying to complete the next token or the or next word, but the kind of things we interact with in the contact center, which is like what dialogue and action on your interact with for someone and then you're working with enterprise software and service of record to like fill out basic things. It's both sort of this intersection of two sets of data sets that don't really exist together, and then that sort of enables us to then train over time these large foundation models that use that intersection of that two data set to sort of do things that aren't possible just with web pages.

18:37Another wedge may be customization. How can you actually use the information from past user behavior to support the user in creating better prompts? Here's Barry's tick. Under the hood, we are doing a ton in terms of constructing the right prompts and parsing responses back from the model APIs we're using. And so again, we have thousands of people already writing SQL and writing Python and doing data work in hex every day. So we have ton of information we've got, we're connected to their database schemas. So we see the structure of their data, we see past queries and past code they've been. So we know which tables and columns are most frequently referenced.

19:13We have information about the project they're building. So we know like, oh, this project is already referencing as part of the schema and that's probably the relevant data for this. You can even look at things like, this is the typical way this organization is formatting their charts and infer how they might want their visualizations to look. And so there's a ton of context we get because we're incorporating this in an existing set of workflows that can help us create the right context and create the right prompt to pass to the model. And I think with all of this right now, again, our focus really is on augmenting the data professionals.

19:45Like this is not like, you know, trying to build some black box thing. It's actually like we have a rule as we're building these features. It's like we don't run the code for you. Like we will generate the code. We will show it to you, but we want to keep the human in the loop. AI sure can feel like a black box. So designing a UI that actually guides the user is also becoming a differentiator, at least for the time being, where you're effectively nudging users to want to interface with these new tools, but also come back. Any learnings in terms of what is increasing completion rates, what is getting people to actually interface with this new, in some ways, super power within the app, because there is the flip side where if it's hallucinating, or if it's not helping them, or in some cases it may even be hindering their workflow, they're not going to return to it.

20:31So how are you thinking about designing that? There's a bunch of things here. I think there's some UI things you've done as an example. Inhex, we kind of have these code cells that you use. And so we put a lot of thought in it. What's the right way to interact with that? It's a comment, completion, I think giving people feedback on what's happening. Some of these models, the latency is still really high. So even something small, like just being really thoughtful on like how you're exposing the user what's going on and why they're waiting. I think one of the things that we've observed to sort of more on the back end in terms of increasing completion rates is when you're building prompts, I think we were tempted early on to try to shove as much context as we could then.

21:10You kind of figure like the more I can tell this model the better. You realize you can pretty easily confuse a model in terms of the amount of context you're passing and so we've been really thoughtful on iterating through and finding what's the right level of context to pass through between what else is going on in the project or the underlying data schemas or yeah it's everything you're doing like before you're sending requests to that API of like building the right context understanding how you're iterating on how different models are going to respond to different types of prompts and different amounts of context how you're potentially even shaming together different types of models or different modalities of models together in one sequence in U .S.

21:47and then how you're giving that feedback to the user. Yeah, I mean I spoke to someone yesterday and what you're saying really resonates with what they said they started off by creating like a lens of like Product, but at the end of the day there wasn't really a moat there because anyone could kind of hook up to dream booth and create the same thing Yeah, right But today I asked him like how he's thinking about differentiating and at least in his scenario He was saying you know, I now link 15 different models and so maybe that isn't much of a moat either But the point is it's not as simple as just hooking into one API So we've touched on different aspects of differentiation from customization to context fetching.

22:24But is any of that truly a moat? Where does value really accrue when your competitors can copy your UI or your latest feature? Let's attack the question head on. Where is the competitive advantage? Where is the moat? Is it in the UI? It feels like that can be replicated very easily. Is it in the cleaning of the data, the linking of the data? Is it something else? Like how are you thinking about the fact that basically you can kind of depend on all your competitors to replicate what you do if it's working. Yeah, we've already seen that. Down to some UI elements that look very familiar. Yeah, the models themselves are commodities and that's going to accelerate even in the next couple of years.

23:05We're going to see a ton of different types of models emerge. We're going to see the costs plunge even further down. I think the ability to plug these in will become as ubiquitous as like using cloud services. It's just like everyone does it being hosted on the cloud is not differentiator in any way these days. I think there's a few places where potential modes emerge. One is we do have a pretty great data advantage. You know, already having hundreds of customers, thousands of users writing millions of lines worth of SQL and Python, we can use that sort of rich information. We're not using it to train models.

23:38I should be very clear. Like that's one thing we've been right in front with our customers about. We're not training models where they would expect to ever have like some code they've written, be a completion for someone else, which is a problem in other places. But it's more like using that actually to personalize for that person and their team. Like, you know, again, like which schema is you're using? What is their code style been like in the past? Like having all of that information and people who are already doing that work in hex gives us a big advantage. Thing two is, and I talk about so the team all the time, like it's kind of what we've already been doing, which is building really thoughtful, well constructed user experiences in UI.