Synthetic Data and the Future of AI | Cohere CEO Aidan Gomez

0:00Would Google have created the ChatGPT moment? I don't think Google had any idea then, otherwise they probably wouldn't have given it away. People were saying Google missed AI wave, but I think they've come back super strong. Like the models that they're building, potentially the best model, they may have surpassed OpenAI. And they have a money printing machine in the back that can fuel it. And they have a data printing machine. And the talent, right? Like the concentration of talent is surreal. They've caught up on a technological basis. The next big question mark is from a product perspective, can they actually compete?

0:31listed as one of the co-authors of what has now become one of the most consequential papers in our industry. What is the core unique insight in that paper that propelled so much of this forward? Efficiency. It was extremely well suited to scaling it up across many GPUs. That was a very important property for the future of machine learning because everyone started scaling up models. And so the models that scaled the best dominated and that turned out to be the transformer.

1:11welcome to grit i'm juvin partner at kleiner perkins a show where we go beyond the highlight reel and explore the personal and professional challenges of building history making companies today on the show we're joined by aiden gomez co-founder and ceo of cohere he is a google brain alum and the co-author of the famous paper attention is all you need cohere is building an enterprise first llm infrastructure and trying to go into the enterprise and regulated sectors like finance and healthcare their company is valued at around 6.8 billion and they are in the eye of all eyes of the ai hurricane enjoy the episode you know i used to travel a lot um i have been more busy on the home front here i would what's that do you have kids not yet but just life has gotten busier here there's a lot of action happening in the bay area you know that's very true yeah yeah it's just a lot going on yeah um i would say you know you're kind of in a sales job right now like the ceo job at this point is the company is kind of a sales job totally i grew up in sales uh and so i like i've been traveling for a long time right as part of that sales yeah you know like that's just that's just the gig yeah uh for a while it's cool and then and then it becomes not cool and people still have this like glorified like oh wow you get to fly every week you know as if it's a good thing and i'm like it's horrible you have like no idea like i wouldn't wish this on my worst enemy yeah and i think it's actually a good hack for you to bring your bring your wife yeah yeah for for as many times as she's willing to but then you like do you feel guilty when you like uh go to like meetings all day and she's alone yeah and then like a work dinner uh well i like just insist that my wife has to come to my no way totally i'm just like she's on this trip with me come you know meet my and they're they're actually stoked about it any meeting that you have any any dinner meeting yeah not like meetings during the day is she in tech no no okay um but i mean by proximity sort of right well by proximity a lot yeah yeah and sorry just so i can be precise about this you'll when you travel you'll be like all right sweetheart i have x xyz trips coming up this is the this is the itinerary this is where i'm going yeah which is instead of the menu of options.

3:43Which one you want to go to. Exactly. And then she picks, you know, something from the appetizer, something from the van, something you deserve. And then when you go, she comes to the dinners with you. Doesn't go to the meetings all day. And then you come back to the hotel, get ready. She gets ready with you. You guys go to dinner together. Yeah, exactly. That's cool. I mean, look, I guess when you're you and the company is uh what was the last like you've raised it whatever six seven seven billion seven billion you're like uh one of the authors of of of the paper that started a lot of this craziness at google attention is all you need i guess there's probably a lot of people that want to sit down with you and if the trade-off is hey like uh you know they have to have dinner with family that seems pretty fair.

4:38I think that's okay. That's actually better. I think they, uh, they usually prefer talking to her than me. Yeah. I mean, she's more interested. Good. Um, well, thanks for doing this. Um, when you, the paper that you wrote, that you co-wrote, I guess you co-authored it, right? With whatever, eight people, nine people at Google. There's eight of us. Yeah. You were an intern. Is that right? Yeah. I was still an undergrad. Yeah. Second, third year undergrad. And I'm like, okay, can you just tell me the story? Like, how did that come to be? Because I'm like, okay, did he like write a few words and then like get his name on there?

5:13Like, I just have a hard time. I can say like, I'm just imagining you at, how old were you? I was 19. 19. Yeah. uh being one of listed as one of the co-authors of what has now become maybe the one of the most consequential papers in our industry totally yeah uh obviously i don't think you had any idea then i don't think google had any idea then otherwise they probably wouldn't have given it away yeah yeah tell yeah well i mean would it have been big if google didn't give it away would it have blown up like this right you're saying a secret you're saying would google have known what like would google have created the chat gpt moment yeah yeah exactly um or would have the you know academic community created a different architecture under a different name that looked very similar and felt very similar that carried the weight instead and so maybe Maybe Google had its proprietary Transformer model, which nobody could use because it was IP and didn't become very popular.

6:20But the rest of the world created something slightly different, which had enough of the same properties to drive all the stuff that we've seen over the past 10 years. I think that's probably more likely. Because at the time, the Transformer was built off of a set of ideas like, what was it called? uh, ByteNet, WaveNet, um, uh, whatever it was, like Seek2Seek. There were these, um, series of papers which had these ideas of auto-aggressive models, which were much more scalable for training. And the ideas were out there in the ether, and we just sort of pulled them together in a particular arrangement.

7:02I think if we hadn't have done it, someone else would of within the next 12 to 18 months. So if Google didn't tell anyone about it, kept it secret, I think someone outside would have created something very similar under a different name. Why do you think that? Because all those ideas were like in the ether. It was just building towards that. Yeah, totally. Totally. Like it was necessary. Like something like the transformer had to be tried by someone. It's just that, you know, we were the ones that tried it first. Um, and we got enough eyes on it to catalyze this snowball. And once it started rolling, once people started.

7:44This is once you create, once you wrote the paper. Yeah. Once we released it, published it at NeurIPS, um, it got enough eyes and momentum that the community figured the rest out, right? Like we sort of like started the seed, like, you know, packed a little snowball and then gave it a little nudge down a hill. and then the community just poured in and started implementing all the different frameworks, you know, testing this thing on all the different benchmarks. And the community carried the work. It was less like us at Google pushing this thing forward, showing the world what it could do. It was everybody else taking this little seed and growing it into what the Transformer is today.

8:25And in your words, like, what is the core unique insight in that paper? that propelled so much of this forward? A focus on efficiency. So it was a very simple architecture, like very minimal. And it was extremely well-suited to scaling it up across many GPUs. At the time we built it, like many GPUs was tens of GPUs, right? Like that was like... And what's many GPUs today? oh tens of thousands you know maybe hundreds of thousands um in a couple years maybe millions but yeah so like the the core insight was hey we're gonna need to scale this thing up and for us that meant instead of training on one gpu we would like to train on 32 um and so let's make sure we build a training framework and an architecture that's compatible uh with that It makes that easy, it makes it efficient.

9:29And then it just so turned out that, you know, that was a very important property for the future of machine learning because everyone started scaling up models. And so the models that scaled the best dominated and that turned out to be the transformer. But at the time, I mean, obviously, how clearly do you even like have the events that have transpired over the last few years, like fuzzied your memory of what actually happened? I have a terrible memory. So, you know, I can't even answer that. I don't know. That's funny. I have certain, like, there are just these really like core memories from that time period.

10:06Like the other thing to remember is that the entire Transformer project happened in, you know, 12 to 16 weeks, about four months. And so everything is compressed into this like one little period in time in 2017.

10:26And yeah, I have moments like the night we submitted to NeurIPS, which was the conference we published the paper at. I slept at the office that night. And a cleaner who was cleaning the offices, I was like in this little phone booth sleeping, opened the door and it hit me in the head. And that woke me up. um and i remember another moment where i was like laying on the couch next to ashish like ashish vaswani like the first author of the paper and we had like you know submitted the paper and i was lying next to him sort of like looking up at him and he's like dude like this is going to be a big deal and i was an intern and this was like my first paper that i was submitting to a conference.

11:12And so I was like, what do you mean? Like, this is, aren't all papers like this, you know? And so I always say that I never had any idea what was coming, but Ashish on that night told me this was going to be a big deal. Now, what I think he was saying was like, we're going to get like a hundred, like hundreds of citations on this paper. You know, this is going to be huge. And at this point it's like hundreds of thousands. So I don't think anyone expected the full scale of the impact. I mean, at this point, people are pointing to that paper as the catalyst for like a next era of an industrial revolution.

11:48Like, yeah, which is kind of insane. Not to over-dramatize it, but that's a pretty big deal. It was a productive four months. No doubt. No doubt. It's a good thing you slept in the office and cranked through some words. From all the things that I have heard, it seems like everybody was caught off guard that these things have been able to scale with more compute and data in the way that they have so far, like for as long as they have. And I wonder, like, is that surprising to you? It totally, yeah. I mean, even around the time Cohere got started, it still wasn't clear. Like there were still papers being written about, like theory papers showing that as you scale models, they should actually get worse.

12:37There's a period where they get better as you scale them up, and then they start getting worse. And some of the big names in the field just didn't have this conviction that bigger is better forever. And so it was a very contrarian viewpoint to say that we could just continue scaling these things up from a model perspective, from a data perspective, and we would continue to get sustained gains. I think even when I started Cohere, I don't know if I really believe that. All I knew is there was still some juice to squeeze and I thought things might taper out and just the scaling pursuit wouldn't be enough.

13:27And I think that's turned out to be true, right? The models, if you believe rumors, GPT-5 is not as big as previous models. It's actually getting small. And they did that because the Orion series, which was much larger, didn't succeed. It didn't make things materially better. And so there's a refocusing instead of just scaling up models and pre-training and data to better data, better training methods, new training methods, instead of just pursuing that. So I think we have squeezed a lot out of that pure scaling play. And it's starting to like saturate at this point. But yeah, like people didn't really, you know, have conviction in that at the time, except for OpenAI.

14:18And they made that crazy bet. And it turned out to be true for way longer than people thought. But now we've started to see that sort of taper off. And so do you think that the GPT-5 model is smaller because internally they just don't think the scaling is going to continue to apply in the same way? I don't know what they think internally. I have no idea. I think that... Like, do you think we're saturating? I do think that for sure on pre-training. Maybe I'll ask you a more simplistic way. the next set of models from Anthropic and OpenAI, if we throw the same amount or double the amount of data and money and GPUs at the problem, yes, well, maybe we'll get, are we going to get diminished returns?

15:03Are we going to get such diminished returns that it doesn't make sense anymore? Like, is it going to be a thing? You have no, like, just, you're guessing, right? So let's just caveat what you're guessing. But you're a pretty good person to guess. like uh is it if if we get half the progress that's still from whatever three and a half to four that's still pretty amazing might still be worth it i'm curious like how you see that how you see that yeah it's an interesting it's an economic question right like if you look at the level of uh spending going on to train these models it is doubling in size or 10xing in size in terms of what you spend to build model economic.

15:49I would say at this stage, we don't have strong evidence that that's the case. The past couple model iterations from the labs that are willing to spend tens of billions of dollars on this, they have largely saturated. People see that. The rate of progress is slowing down considerably. and they're not slowing down on spend and scale. And so like purely looking at, you know, one model generation to the next over the past couple cycles, we haven't seen big jumps come through. And for the core businesses, like for the consumer chatbots, I don't think the consumer can feel a material difference. it's not making it so much better for them that they're willing to pay 10 times as much per month to access that model so we are in that respect entering into sort of uneconomic territory now there's a that's the consumer side of things there's the enterprise side of things there's the science side of things, right?

17:06Like what is curing cancer worth, right? Like what, what is the sort of much higher level scientific breakthrough, um, type pursuits worth and how much are you willing to pay to unlock that? Maybe there it becomes economic, right? So, uh, I think a lot of labs are refocusing around, you know, LLMs for science to drive that sort of innovation forward under the hope that governments or extremely large entities will be willing to pay pretty much anything to get access to the outputs of those those models okay you're traveling a lot you're building cohere which we should talk about but like in the meantime how much are you keeping up to speed on like okay you wrote the first paper like how academic are you still these days on the things that are happening?

18:02Dude, I'm non-technical. I'm very, I'm cooked. You think you're done? Yeah, I think I'm more annoying to the modeling team at Cohere than I am helpful. And I'm being self-deprecating, but I keep up with the papers that I find extremely interesting. I used to read just out of the fire hose, right? Anything that was coming out, I would consume it religiously. at this point I read a paper like I don't know once a quarter type thing twice a quarter and it's only if it's super interesting you know there's people around me there's like fantastic researchers on the Cohere team who will like surface things to me like Aiden you need to see this so I have a really turning into a sales guy dude it's you're so good but I mean it's like it's a privilege as well right like we were talking about I get to I get to go to really cool places.

19:02I get to meet really cool people. And being involved in a different side of the field, right? Like I used to be a researcher, but now I get to help set policy, help actually deploy the technology into the global economy. get to see that sort of at the front line in a way that when i was in oxford or you know when i was in mountain view or toronto uh doing research i was so far from that shit i was so far from anyone actually using the stuff that i was uh i was building um so no it's just a it's a different job it's a different job i do miss research though and i am annoying to the cohere modeling team i always have ideas you know we should be trying this we should what kind of ideas like technical breakthrough ideas like things that efficiency ideas yeah stuff like that you know like i don't know like um multi-hot prediction uh lots of stuff about like can we train models low rank to get much better uh you know efficiency out of these things and like get more out of the training compute can we make pre-training more efficient um data curriculums right like first train on the easy stuff and then trade on harder stuff, train on the noisier stuff first and cleaner stuff.

20:19I have so many ideas on how to make this shit more efficient just because that's where I came from. Then of course, I guess the one place that I still do contribute are the meta or the product level modeling questions. So one of those is, you know how reasoning models, I guess this time last year, came out? And since then, they've just unlocked a category of problem that previously you could not solve with LLMs, and now you can. Like complicated math, a lot of coding debugging needs reasoning. You need to understand why that first attempt failed, fix that tool use inside of the enterprise, like using all these enterprise systems, something might fail.

21:11One attempt at accomplishing something might fail, and you need to reroute and figure out how to solve that. And that requires reasoning to do that. And reasoning was so obvious. It surprised people who I think were one step removed from language models. But good God, the premise of a language model where you have an input stream, which is like a series of words, that input space is everything. Those words could be any question. It's something as simple as like, what's one plus one, which should be quite easy to answer. but also like a question, you know, how do we solve this theorem, like this millennium problem, or how do you know, what is the cure to cancer, right?

21:59Like the problem you present the model with varies in complexity massively. And yet, language models pre-reasoning would spend the same amount of compute answering those two massively different levels of complexity. Whereas with humans, like we spend different, you know, some problems we work on for decades and some problems we can answer immediately. So reasoning gave us that variability of compute or effort to solve varying complexities of problem. But there are other very obviously missing capabilities in these models that humans have. So one of those is learning from experience. Like when I, you know, Cohere is like my first company.

22:49When I founded Cohere, I had zero experience in founding a company. And so I f***ed up everything, you know, like every mistake you could make building a company, I promise you I've done. But the good thing about being a human is you learn from that experience. You get a little bit better that first f*** up. Hopefully you don't make it more than once. And you grow over time, you become massively more competent. LLMs, if I spend a month working with an LLM as a sort of colleague with me, and then I press new chat, it's reset to that same LLM that I was speaking to at the beginning of that month.

23:31And so it's like that's an obvious trait of intelligence that we're missing. So I think that will be the next thing that comes out. And that's the type of thing that I push the team on is like, we're missing this. Yeah. Think of what this unlocks at the product level. Yeah. We need it. Is that like a, is that like a memory problem? Like, is that like a recall of memory problem? It is. Yeah, for sure. For sure. Memory is a huge component of that. It's also learning skills. So distilling out of memories, distilling out of experience, uh, abstractable, um, skills that you can apply elsewhere. And I guess if you believe that that's like the next shoe to drop in the space that you think is as obvious as reasoning was whatever 12 or 18 months ago, then you really don't even need any, we could stop at the models today.

24:26Like you don't need any more advancement to be able to improve memory and recall and some of the things that you're describing, right? um so are you saying we don't need a new architecture i'm saying like if we stopped at gpt5 right like we don't need it we don't need gpt6 in order to improve this specific problem that you're describing well i think gpt6 is the solution to the you know like the next major update to language models will be this capability you think so absolutely Absolutely. And what will that mean for the consumer and for the enterprise? It means it will get smarter over time without you having to go train a new model.

25:15It'll learn more about you. It'll learn more about, you know, for Cohere, it'll learn more about the enterprises that we work with. Like initially the model shows up and it's like an intern out of school. It has like a bunch of general experience. Universities and schools are fantastic at giving generic knowledge about a field. But when that intern shows up, they completely lack the specific knowledge of your business, your product, your customers, etc. But over time, that intern becomes more experienced, gets a full-time offer, has been there for a few years, and they're dramatically more productive and effective for you.

25:51And so just the fact that they've been sitting there running, seeing your business, they become more valuable to you. that's the product unlock is that with more time and more use this asset becomes increasingly high roi to you and is the argument that like i think open ai planted their flag in the ground that around i think around this time next year maybe their goal is to have an ai researcher and um maybe like is if the path is very obvious to gpt6 being this key unlock where the system self-learns much more efficaciously than it used to right similar to how a human does then does it get to the path where it can also start to train itself like uh does it start to get to the path where if it starts to learn from its own mistakes, like even today or yesterday, I think I saw something in a paper about signs of self-awareness in Claude's model.

26:58But if it starts to become self-aware, whatever. You disagree with it? You think that's BS? Yeah, I think it's BS. Why? I don't know. The effective altruist crowd loves to personify these things. And you think it's totally stupid? Yeah, I think. Yeah. Yes, I do. I want to revisit that in a second, but is that the path to the model doing a lot of its heavy lifting on its own? Yeah, I mean, to self-improvement, already there's a lot of self-improvement, actually, just to be clear. Synthetic data, people did not believe in synthetic data a couple years ago. They were saying you can't learn it. It's like an Ouroboros.

27:39It's like a human centipede of data. It's like if the model is producing its own data, it can't make itself smarter. there's some sort of limit there or degradation that's going to take place but now across every single lab the majority of the data is synthetic there's so much to be won by having these models reformat their own data in a way that lets them learn more effectively that filters for better data structures things in a better way so already these models are sort of teaching themselves or helping themselves learn. The AI scientist thing from OpenAI, I don't know, let's see. There's been a lot of this meta-learning stuff where you get models to tweak hyperparams and do search.

28:37It's all sort of been a very expensive waste of time. You seem a little annoyed by the model companies. Let me just put it that way. Maybe like... here is a model company well okay you seem a little annoyed by the bluster of what a lot of these labs are saying is that a fair characterization that's a fair characterization i think the uh whole like agi asi doomsday stuff like the goalposts keep moving right it was like we can't release uh you know gbt3 because it literally might kill the world you know like That was the sort of ethos that was out there. This is so dangerous. To lock this down, let's bomb data centers of our adversaries so that they can't build models.

29:24And also, roll back two years. You have folks, I won't name names, but you have folks who are saying there is going to be just an exponential takeoff of model capability because having the smartest model means it's easier to build the next smartest model and it'll just run away and the first person to get there takes the whole cake. And in reality, look at what has happened over the past couple of years. We have like five to seven models that have all converged to the same fixed point. They're all like interchangeable. There's barely any different. Not only has someone else caught up with a lot of resources, like seven have.

30:10And so there were all these extraordinary claims and posturing made, which we've just seen over time repeatedly proved to be demonstrably false. And it feels like that bravado, that sort of posturing did a real disservice to the world. And I think it was posturing to scare others off. from trying. You'll never be able to catch up. We're already ahead. There's no way that you can do it. Or saying how much money or energy you need in order to... Yeah, exactly. Or going to the government and saying, we have to control this. No one should be allowed except for me and my buddies to train something about this.

30:56You think it's pulling the ladder up behind you. Totally. Totally. And I think that that was a highly effective strategy. I think it scared off a lot of people. I think it scared investors, it scared this and the other, it scared regulators, policy people. It was a hyper-effective strategy of pulling up the ladder and disincentivizing people to participate and try and play. It was extremely intellectually dishonest, which I think is what hurts the most. Because these were claims about the technology. And these were very unreasonable extrapolations. And so if you've heard them for half a decade espousing this, pushing this with so much conviction, so much confidence, you know there is a certain i just don't i don't like that i'm kind of i'm kind of done with that phase of like the world should be afraid of this technology you know this this technology is going to destroy the world i think the technology could save the world right like i think this technology is so high leverage across so many different fields we shouldn't be scared of it we should be running towards it we should be deploying it as quickly as we can.

32:25And I think the world has broadly accepted that now. But a year ago, two years ago, it was a very different story. Yeah. You know, the thing that we're hearing now is, oh, well, just wait until it gets into robots. Like that's when things are going to be really scary, you know? And so it seems like the pressure points of fear just shift into other areas, right? Yeah, fear is motivating. If you can catalyze fear in a direction that's productive to you, it's a super effective strategy. I think it's a really ugly strategy. I don't like it, but it's proven quite effective in AI. Can you give the 30 seconds on Cohere?

33:06Yeah, sure. We're a modeling company. We build models. We also build the framework and platform around the models to deploy them into businesses. If you think about what a model needs to be and productive inside of a company, it needs the same level of access that you grant, trust and access that you grant to your humans inside the organization. So it needs to be able to use your sales software, HR software, marketing software, your email, your calendar, your supply chain software, everything. And it needs to be able to use it in the same way a human uses it. And so that's what we built. That's our North platform.

33:45It integrates into, We focus mostly on what I would describe as critical industries, public sector, energy, financial services, healthcare, telco. Anywhere that data is a national security concern, and these are sort of the foundations of a country's economy, Cohere does extremely well. principally because our deployment model is different than the other guys where instead of deploying only on cloud yes we can deploy on all clouds but we can also go on print and so for extremely secure settings where maybe you even want to go air gapped our models run just fine and so they're designed for these very private strict environments from scratch we like set a constraint saying the model we build, we're not just going to build the biggest model because like the enterprise market cannot consume the biggest model.