Model Plateaus and Enterprise AI Adoption with Cohere's Aidan Gomez

0:05Hi, listeners, and welcome to KnowPriors. Today, we're hanging out with Aidan Gomez, co-founder and CEO of Cohere, a company valued at more than 5 billion in 2024, which provides AI-powered language and no-powered. Aidan founded Cohere in 2019, but before that, during his time as an intern at Google Brain, he was a co-author on the landmark 2017 paper, Attention is All You Need. Aidan, thanks for coming on today. Yeah, thank you for having me. Excited to be here. Maybe we can start just a little bit with the personal background. How do you go from growing up in the woods in Canada to, you know, working on the most important technical paper in the world?

0:45A lot of luck and chance. But yeah, I happened to go to school at the place where Jeff Hinton taught. And so obviously Jeff recently won the Nobel Prize. He's kind of like attributed with being the godfather of deep learning. In U of T, the school where I went, he was a legend and pretty much everyone who was in computer science studying at the school wanted to get into AI. And so in some sense, I feel like I was raised into AI. Like as soon as I stepped out of high school, I was steeped in an environment that really saw the future and wanted to build it. Um, and then from there, it was a bunch of happy accidents.

1:35So I, I somehow managed to get an internship with Lukash Kaiser, uh, at, at Google brain. Um, and I found out at the end of that internship, I wasn't supposed to have gotten that internship. It was supposed to have been for PhD students. And so they were like throwing a goodbye party for me, the intern. And Lukash was like, OK, so Aiden, you're going back. How many how many years have you got left in your Ph.D.? And I was like, oh, I'm going back into third year undergrad. And he was like, we don't do undergrad internships. So I think it was a bunch of like really lucky mistakes that led me led me to that team.

2:26Working on really interesting, important things at Google, what convinced you that you should start Cohere? So I bounced around like when I was working with Lukash and Noam and the Transformer guys, I was in Mountain View. And then I went back to U of T, started working with Hinton and my co-founder Nick in Toronto. I brained there. And then I started my PhD and I went to England. And I was working with Jakob, who's another Transformer paper author in Berlin. We had Jakob on the podcast. Oh, nice. Yeah, yeah, yeah. Okay. Fan of the pod. Good, good. So yeah, I was working with Jacob in Berlin. And then I was also collaborating remotely with Jeff Dean and Sanjay on Pathways, which was like their, you know, bigger than a supercomputer training program.

3:19The idea was like wiring together supercomputers to create a new larger unit of compute that you could train models on. And at that stage, GPT-2 had just come out. And it was pretty clear the trajectory of the technology. Like we were on a very interesting path. and these models that were ostensibly models of the internet, models of the web, were going to yield some pretty interesting things. So I called up Nick, I called up Ivan, my co-founders, and I said, you know, maybe we should figure out how to build these things. I think they're going to be useful. For anyone who doesn't know yet, can you just describe at the high level, like what Cohere's mission is and then what the models and products are?

4:11Yeah, so our mission, the way that we want to create value in the world is by enabling other organizations to adopt this technology and make their workforce more productive or transform their product and the services that they offer. So we're very focused on the enterprise. We're not going to build a chat GPT competitor. What we want to build is a platform and a series of products to enable enterprises to adopt this technology and make it valuable. And in terms of like your North Star of how you organize the team and invest, you obviously come from a research background yourself. Like how much do you think, you know, Cohere's success is dependent on core models versus other platform and go to market support investments you make?

4:59It's all of the above. Like the models are the foundation. And if you're building on a foundation that doesn't meet the customer's needs, then there's no hope. And so the models are crucial and it's like the heart of the company. But in the enterprise world, things like customer support, reliability, security, these are all key. And so we've heavily invested on both sides. We're not just a modeling organization. We're a modeling and go-to-market organization. And increasingly, product is becoming a priority for Cohere. And so figuring out ways to shorten time to value for our customers. Yeah, over the past 18 months since the enterprise world sort of woke up to the technology, we've watched folks build with our models, seeing what they're trying to accomplish, seeing the common mistakes that they make.

5:58That's been helpful. It's been sometimes frustrating, right, watching the same mistake again and again. But we think there's a huge opportunity to be able to help enterprises avoid those mistakes and implement things right the first time. And so that's really where we're pushing towards. Yeah. Can we make that a little bit more real? Like what is the mistake that frustrates you most and how can product go meet that? Yeah, well, I think all language models are quite sensitive to prompts, to the way that you present data. They all have their own individual quirks. The way that you talk to one might not work for the way that you talk to another.

6:34And so when you're building a system like a RAG system where there's an external database, it really matters how you present the retrieved results to the model. It matters how the data is actually stored in those databases. The formatting counts. And these small details are often lost on people. They overestimate the models. They think they're like humans. And that has led to a lot of repeat failures. People try to implement a RAG system. They don't know about these idiosyncratic elements of implementing one properly. And then it fails. And so in 2023, there were a lot of these POCs, a lot of people trying to get familiar with the technology, wrap their heads around it.

7:26And a lot of those POCs fail because of infamiliarity, because of, yeah, these common errors that we've seen. And so moving forward, we have two approaches. One is making the models more robust. So the model should be robust to a lot of different ways that you present data. And the second piece is being more structured about the product that we expose to the user. So instead of just handing a model and saying, you know, prompt it, good luck. Actually putting more structure around it. So creating APIs that more rigorously define how you're supposed to use the model. These sorts of pieces, I think, just reduce the chances of failure and make these systems much more usable for the user.

8:13What are people trying to do? Can you give us a flavor of some of the biggest use cases you see in the enterprise? It's super broad. So it spans pretty much every vertical. I mean, the common things are like Q &A. So speaking to a corpus of documents. For instance, if you're a manufacturing company, you might want to build a Q &A bot for your engineers or your workers who are on the assembly line and plug in all of the manuals of the different tools and diagnostic manuals for common errors in parts. And then let the user chat to that instead of having to open up a thousand page book and try to find what they need.

8:55Similarly, Q &A bots for the average enterprise worker. So plugging in your IT FAQ, your HR docs, all the things about your company, and having a centralized chat interface onto the knowledge of your organization so that they can get their questions answered. Those are some of the common ones. Beyond that, there are kind of specific functions that we power. A good example might be for a healthcare company, they have these longitudinal health records of patients. And that consists of every interaction that that patient has with the healthcare system, from visits to a pharmacy, to the different labs or tests that they're getting, to doctor's visits.

9:44And it can span decades. And so it's a huge, huge record of someone's medical history. And typically what happens is that patient will call in and they'll ring up the receptionist and be like, my knee hurts. I need an appointment. And the doctor then needs to kind of comb through the past few entries. He has this come up before and maybe they missed something that was two years ago because they only have 15 minutes before an appointment. But what we can do is we can feed that entire history in alongside the reason they're coming in. So contextually relevant, right, to what they said they're coming in for and surface a briefing for the doctor.

10:27and so this tends to be one dramatically faster for the doctor to review but also often it catches things that a doctor couldn't possibly review before every patient meeting they're not going through 20 years of medical history like it's just not possible but the model can do that it can do that in under a second so those are the sorts of functions that we're seeing summarization Q &A bots, a lot of these, you might think of them as mundane, but the impact is immense. We see tons of startups working on problems such as, let's say, enterprise search overall, specialized applications to, let's say, like technical support for a particular vertical, even looking at health records and reasoning against them and retrieving from them.

11:16How do you think about like what the end state, there's no end state, but what some stable equilibrium state is for how enterprises consume from, let's say, specialist AI powered application providers versus custom applications built in house with AI platforms and model APIs? I think it's going to be a hybrid. I think it's probably you can imagine like a pyramid where the bottom of that pyramid, every organization needs this stuff. And it's like co-pilot, like a generalist chat bot in the hands of every single employee to answer their questions. And then as you head up the pyramid, it's more specific to the company itself or the specific domain or product that they operate in or offer.

12:04And as you push up that pyramid, it's much less likely you're going to find an off-the-shelf solution to address it. And so you're going to have to build it yourself. What we've pushed organizations to do is have a strategy that encompasses that full pyramid. Yes, you need the generalist standard stuff. Maybe there's some industry specific tools that you can go out and buy. But then if you're building, don't build those things that you could buy. Instead, focus on the stuff that no one's going to sell to you. And that gives you uniquely a competitive advantage. So we worked with this insurance company and they insure like large industrial development projects.

12:46It turns out, I know nothing about this space. It turns out what they do is there's like an RFP put out by a mine or something, like whatever the project is for insurance. And they have actuaries jump on that RFP, do tons of research about, you know, the land that it's on, the potential risks, etc. And then it's essentially a race to whoever responds first usually gets it. And so it's a time-based thing. How quickly can these actuaries put forward a good researched proposal? And what we built with them was like a research assistant. So we plugged in all the sources of knowledge that these actuaries go to to do their research via RAG.

13:32And we gave them a chatbot. and it dramatically sped up their ability to respond to RFPs. And so it grew their business because they were just winning many more of them. And so it's tough for like, we built horizontal technology and LLM is kind of like a CPU. I don't know all the applications of an LLM, right? It's so broad and really the deep insight or the competitive advantage, the thing that puts you ahead is listening to the customer and letting them tell you what would put them ahead. And so that's a lot of what we've been doing is just being a thought partner and helping brainstorm these projects and ideas that are strategic to them.

14:18I'd wager that this company is winning because the vast majority of their competitors haven't been able to move so quickly to adopting and building like this research assistant product that is helping them. Like what is the biggest barrier you see to generally enterprise adoption? I think the big one is trust. So security is a big one. In particular, in regulated industries like finance, healthcare, data is often not in a cloud, or if it is in a cloud, it can't leave their VPC. and so it's very locked down. It's very sensitive. And so that's a unique differentiator of Cohere. The fact that we haven't locked ourselves into one ecosystem and we're flexible to deploy on-prem if you want us, in VBC, outside of VBC, literally whatever the customer wants, we're able to touch more data, even the most sensitive data and provide something that's more useful.

15:23So I would say security and privacy is probably the biggest one. Beyond that, there's knowledge, right? Like the knowledge to know how to build these systems. They're new. It's unfamiliar to folks. You know, the people with the most experience have a few years of experience. And so that's the other major piece. that bit, I think it's honestly just a time game. Like eventually developers will become more familiar with building with this technology. But I think it's going to take another two or three years before it really permeates. Do you think in like a traditional hype cycle for enterprise technologies, probably for most technologies, but in particular enterprise, you know, there's this trough of disillusionment concept of people get very excited about something and ends up being harder to apply or more expensive than they thought.

16:18Do we see that in AI? I'm sure we see some of it, for sure. But I think, honestly, like the core technology is still improving at a steady clip. And new applications are getting unlocked every few months. So I don't think we're in that trough of disillusionment yet. Yeah, it feels like we're super early. It feels like we're really, really early. And if you look at the market, this technology just unlocks an entire new set of things that you couldn't build. You just fundamentally couldn't build them before and now you can. And so there's a resurfacing of technology, products, systems that's underway.

17:00Even if we didn't train a single new language model, like, okay, all the data centers blow up. We can't improve the LRM. We only have what we have today. There's a half decade of work to go integrate this into the economy, to build all these things, to build the insurance RFP response bot, to build the healthcare record summarizer. There's a half decade of just resurfacing to go do. So there's a lot of work ahead of us. I think we're kind of past that point. There was a question of, oh, is there too much hype? Is this technology actually going to be useful? But it's in the hands of 100 million people now, hundreds of millions of people now.

17:44It's in production. There's very clear value. The project is now putting it to work and delivering it to the world. In this question of like integration into the real world, some piece of it is, of course, like interfaces and change management and like figuring out how users are going to understand the model outputs and guardrails and all of that. Specifically, when we think about the model and specialization, like, do you have some framework you offer customers or that you use internally around what version of it they should invest in, right? So we have pre-training, post-training, fine-tuning, retrieval, like in the sort of traditional sense, like prompting, especially as we get longer context, like how, how do you tell customers to make sense of how to specialize?

18:36It really depends on the application. Like there's some stuff, for instance, we partnered with Fujitsu, um, who's like the largest SI in Japan, um, to build a Japanese language model. There's just no way you can do that without intervening on pre-training. You can't like fine tune or post-train Japanese into a model effectively. And so you have to start from scratch. On the other side, there's more narrow things. Like if you want to change the tone of the model or you want to, I don't know, change how it formats certain things, I think you can just do fine tuning. You can take the end state. And so there is this gradient.

19:19What we usually recommend to customers is start from the cheapest, easiest thing, which is fine tuning, and then work backwards. And so start with fine tuning, then go back into post training, right? Like SFT, RLHF. Then if you need to, and you know, it's kind of a journey, right? Like as you're talking about a production system and the constraints are getting higher and higher, you potentially will need to touch pre-training. Hopefully not all of pre-training. Hopefully it's like 10 % of pre-training at the very end, or maybe 20 % of pre-training. But yeah, that's usually how we think about it is like this journey from the simplest, cheapest thing to the most sophisticated, but most performant.

20:00Moving along the gradient from the cheapest thing makes sense to me. The idea that any enterprise customer will invest in pre-training is, I think, a bit more controversial. I believe some of the lab leaders would say like nobody should be touching this and it doesn't make any sense for people from a scale of compute and data curation effort required and just sort of the talent required to do pre-training in any sort of competitive way? Like, how would you react to that? I think if you're building like a, if you're a big enterprise and you're sitting on a ton of data, like hundreds of billions of tokens of data, pre-training is a real lever that you're able to pull.

20:44I think for most like SMBs and certainly startups, it makes no sense that you should not be pre-training a model. but if you're a large enterprise I think it's it should be a serious consideration the question is how much pre-training it's not like you have to start from scratch and do a $50 million training run but you can do a$5 million training run that's what we've seen succeed these sort of continuation pre-training efforts so yeah that's one of the offerings that we have. But of course, we don't jump straight into that. You don't need to spend massively if you don't want to. And usually the enterprise buying cycle or technology adoption cycle is quite slow.

21:34And so you have time to move back into it. I would say it's totally at the customer's discretion. But to the folks who say that no one should be pre-training. No one outside of, let's say, AGI labs should be pre-training. That's empirically wrong. Maybe that's a good jumping off point into just talking a little bit more about what's going on in the technical landscape and also what that means for Cohere. What is the bar you set internally for Cohere? You said the models of the foundation. And I believe you've also said there's no market for last year's models. How do you square that with the expense of, the capital expense of competition and the rise of open source models now?

22:16Well, I think you have to spend, there's some like minimum threshold that you need to be spending at in order to build a model that's useful. The things get cheaper. The compute to train the model get cheaper. The sources of data, well, in some directions they get cheaper and others not. With synthetic data, it's gotten dramatically cheaper. But with expert data, it's getting harder and harder and more expensive. And so what we've seen is today you can build a model that's as good as gpt4 in all the things that enterprises might care about for 10 million dollars 20 million dollars like just orders of magnitude less than what was spent to develop that model and so if you're willing to wait six months or a year to build the technology you can build it at a fraction of what those frontier labs have paid to develop it.

23:15And so that's been a key part of Cohere's strategy is we don't need to build that thing first. What we'll do is we'll figure out how to do it dramatically cheaper and we'll focus on the parts of it that matter to our customers. So we'll focus on the capabilities that our customers really depend on. Now, at the same time, we still have to spend like relative to a regular startup. We have to pay for a supercomputer and those things cost hundreds of millions of dollars a year. So it is capital hungry, but it's not capital inefficient. It's very clear that we'll be able to build a very profitable business off of what we're building.

23:57So that's the strategy is don't lead, don't burn, you know, three, five, seven billion dollars a year to be at the front, be six months behind and offer something to market to enterprises that actually fits their needs at a price point that makes sense for them. Why spend on the supercomputer and the training yourself at all if you have increasingly open source options? Well, you don't, not really. Say more. So for Llama, yeah, you get like the base model at the end when it's cooled down and it has zero gradient. You get the post-trained model at the end when it's cooled down and has zero gradient.

24:39Taking those models and trying to fine tune them, it's just, it's not as effective as building it yourself. And you have much fewer levers to pull than if you actually have access to the data and you can change the data that goes into that process. And so we feel that by being vertically integrated and by building these models ourselves, we just have dramatically more leverage to offer our customers. Maybe if we go to projection and we'll hit on a few things that you've mentioned as well, where are we in scaling loss? Like how much capability improvement do you expect over the next few years? We're pretty far along, I would say.

25:23Like we're starting to enter into a sort of flat part of the curve. And we're certainly past the point where if you just interact with a model, you can know how smart it is. Like the vibe checks, they're losing utility. And so instead, what you need to do is you need to get experts to measure within very specific domains like physics, math, chemistry, biology.

25:51You need to get experts to actually assess the quality of these models because the average person can't tell the difference at this stage between generations. Yes, like there's still much more to go do, but those gains are going to be felt in very specialized areas and have impacts on more researchy domains. I think for enterprises and the general sorts of tasks that they want to automate or tools that they want to build, the technology is already good enough or close enough that a little bit of customization will get them there. So that's sort of the stage that we're at. There's a new unlock in terms of the category of problems that you can solve, and that's reasoning.

26:38And so online reasoning is something that has been missing. These models, they previously didn't have an internal monologue. right like they didn't really think to themselves you would just ask them a question and then expect them to immediately answer that question they couldn't reason through it they couldn't fail right like make a mistake catch that mistake fix it and try again and so the fact that we now have reasoning models coming online of course open ai was the first to put it into production but cohere has been working on it uh for about a year now um this category of tech i think is is really interesting.

27:13There's a new set of problems that you can go solve. Um, and it also changes the, it changes the economics. So before, if I had a customer come to me and say, uh, Aiden, I want your model to be better at X, or I want a smarter model. I would say, okay, you know, give us six to 12 months. We need to go, uh, spin up a new training run, train it for longer, train a bigger model, et cetera, et cetera. Now that was kind of the only lever we had to pull to improve the performance of our product. There's now a second lever, which is you can charge the customer more. You can say, okay, let's spend twice as many tokens or let's spend twice as much time at inference time and you'll get a smarter model.

28:02So there's a much nicer product experience. Okay, you want a smarter model? well, you can have it today, you just need to pay this. And so they have that option. They don't need to wait six months. And similarly for model builders, I don't need to go double the size of my supercomputer to hit a requisite intelligence threshold. I can just double the amount of inference time compute that my customers pay for. So I think that's a really interesting structural change in how we can go to market and what products we can build and what we can offer to the customer. I agree. I think it's perhaps undervalued in the ecosystem right now how much more appealing it should be to all types of customers that you can move from a like a CapEx model of improvement to a consumption model of improvement.

28:54Right. And it's not like, you know, these are apples and oranges things. But but I think you'll see people invest a lot more in, you know, solving problems when they don't have to pony up for a training run and have this delay, as you described. Yeah, it hasn't been clocked. Like people haven't really priced in the impact of inference time compute delivering intelligence. There's loads of consequences, even at like the chip layer, right? Like what sort of chips you want to build, what you should prioritize for data center construction. if we have a new avenue which is inference time compute that doesn't require this densely interconnected supercomputer it's fine to have nodes you can do a lot more locally and less distributed i think it has loads of impact up and down this chain and it's a new paradigm of what these models can do and how they do it you were dancing around this but because are, you know, your average person doesn't spend that much time thinking about like, what is reasoning, right?

29:58Do you have any intuition you can offer people for like, what are the types of problems this allows us to tackle better? Yeah, I think any sort of multi-step problem. There's some multi-step problems you can just memorize, which is what we've been asking models to do so far. Like solving a polynomial, right? Like really that should be approached multi-step. That's how humans solve it. We don't just get given a polynomial and then boom. There's a few that maybe we've memorized, right? But by and large, you have to work through those problems, break them down, solve the smaller parts, and then compose it into the overall solution.

30:37And that's what we've been lacking. We've really lacked, and we've had stuff like chain of thought, which has enabled that, but it's sort of like a retrofitting. It's sort of like we train these models to just memorize input-output pairs, and we found a nice little hack to elicit the behavior that mimics reasoning. I think what's coming now is from scratch, the next generation of models that is being built and delivered will have that reasoning capability burnt into it from scratch. And it's not surprising that it wasn't there to begin with because we've been training these models off of the internet.